AlphaSignal

The gateway to the AI frontier.

[AlphaSignal @AlphaSignalAI]

Kimi K3 is getting called Fable/Sol level, and it's 7th in our tests.

Arena Frontend Code: #1 at 1679 points.
Artificial Analysis: #3 at Intelligence Index of 57.

We ran it the next day on our coding-agent repair harness against GPT-5.6 Sol, Fable 5, Grok 4.5, Opus 4.8, GLM-5.2, and Gemini 3.1 Pro.

Results:
> Last of 7 models
> 53 of 67 attempts (79%)
> $0.186 per successful fix
> 702s average wall time

Sol hit 100% (70/70) on the same suite. Grok sat at 99% and 46s.

So why does the internet sound so sure K3 is crushing coding agents, if our tests have it at the bottom?
-----
> Full write-up: https://t.co/B0ae9dbP5A
> 5-min daily signals: https://t.co/ZGatS9hxp5

[Artificial Analysis @ArtificialAnlys]

Kimi K3 scores 57 on the Artificial Analysis Intelligence Index. Its intelligence is comparable to Opus 4.8 and GPT-5.5 but remains behind Fable 5 and GPT-5.6 Sol. Moonshot AI has expressed plans to release the 2.8T parameter model's weights, which would make it the leading open weights model

Key results:

➤ Strong agentic task performance: @Kimi_Moonshot's Kimi K3 reaches an Elo rating of 1668 on GDPval v2. This is a marked improvement over K2.6’s 1190, surpassing GLM-5.2 (1514), GPT-5.5 (1494), and Claude Opus 4.8 (1600). However, it still lags behind Claude Fable 5 (1760). Kimi K3 also scores an impressive 53% and takes the #1 position on AutomationBench-AA, our implementation of Zapier’s Agentic SaaS workflow evaluation.

➤ Second-highest performance on AA-Briefcase (agentic knowledge work): On our private long-horizon knowledge work evaluation, Kimi K3 reaches an overall Elo of 1547, +732 points from Kimi K2.6 and behind only Claude Fable 5. It is well-rounded: its rubric scoring and analytical quality almost reach Claude Fable 5’s scores, while GPT-5.6 Sol continues to outperform other leading models on presentation quality.

➤ Set to lead open weights models once weights are released: Moonshot AI has not yet released the weights but expressed plans to do so. Once available, Kimi K3 would clearly lead other open weights models including GLM-5.2 (51) and DeepSeek v4 Pro (44). However, at 2.8T parameters, it is significantly larger than its open weights peers (eg. GLM-5.2 at 753B params and DeepSeek V4 Pro at 1.6T), as well as the Kimi K2 to K2.6 models (1T params).

➤ Cost per task ($0.94) is similar to GPT-5.6 Sol ($1.04), ~1/2 the price of Opus 4.8 ($1.80) and higher than open weights peers: Moonshot AI’s pricing for K3 is significantly higher than their K2 pricing (K3’s output token price is $15/1M tokens while K2.6 was $4). This positions the model as cheaper on a cost per task basis than Opus 4.8, similar to GPT-5.6 Sol ($1.04) and more expensive than open weights peers, GLM-5.2 ($0.32) and DeepSeek V4 Pro ($0.04)

➤ Improved token efficiency alongside higher intelligence: Kimi K3’s token usage on the Artificial Analysis Intelligence Index decreased significantly, using 21% fewer output tokens than K2.6. The new model used approximately 132M output tokens to complete all nine evaluations, compared to approximately 166M for K2.6, while achieving higher scores.

➤ Native multimodal capabilities: Kimi K3, like K2.6, is released with native image and text multimodal input. If weights are released, this will position Kimi K3 as one of the leading open weights models with multimodal input capabilities

Other model details:

Context window: 1M

Size: 2.8T total parameters

Pricing: The first-party API is priced at $3.00/$15.00 per 1M input/output tokens, with cached input discounted 90% to $0.30 per 1M tokens.

Modality: Native multimodal input supports text and images, and the model remains text-only for output.

Accessibility: Accessible at launch through Moonshot’s first party API. Model weights are not yet released but Moonshot AI has expressed plans to do so.

[AlphaSignal @AlphaSignalAI]

We're presenting the SF AI Assistant Meetup with @merit_systems on October 19.

​This event is a space for power users and people interested in the world of AI Assistants to hang out, get to know one another, and share tips/tricks on how to best use these tools.

RSVP: https://t.co/ytkrqW4HdU

[David Pawlan @DavidPawlan]

SF AI Assistant lovers, your time has come!

October 19th. Get ready for an epic AI assistant meetup 🤖

To be announced:
> more sponsors/partners
> location
> event agenda

RSVP here: https://t.co/F8F2V3lO14

[AlphaSignal @AlphaSignalAI]

Those 20 times are 22 GPU-hours against 1.14. Throughput on one 5090 was about 8.4K tokens/s against 124K.
Full Breakdown: https://t.co/uRInSRHlNA

[AlphaSignal @AlphaSignalAI]

You’ve probably heard the buzz around Looped transformers, but which one?

Over the past month, the same word covered three reuse tricks.

Implementing the wrong one burns a training week. It also budgets a 3B file like a 3B decode.

We opened a looped transformer open model, Nanbeige 4.2:

> 22 layers, two passes
> 44 applications, same weights
> About twice the block compute
> KV cache doubles with it

It kept about 75% of a standard Transformer's token efficiency.

Sharing the cache halved its memory. And quality gains were consistently lower.

A second design carries state token to token. Sequence length is the depth.

Training used 500 million tokens. That run cost about 20 times the GPU-hours.

So, Which reuse trick is the viral tweet?
If extra loops do not raise quality, what are you paying?
Does the file still need a second cache?

Full Breakdown ↓↓

[AlphaSignal @AlphaSignalAI]

@LiorOnAI Solve it already.

[Lior Alexander @LiorOnAI]

It's time to solve one of the biggest challenges in vision-language models.

Today's multimodal models can look at a biomedical image and give a convincing, sometimes correct, answer without actually understanding what they're seeing.

A model might answer the medical question correctly while failing to identify basic visual context like the imaging modality, body part, specimen, or stain.

So, I'm teaming up with Stanford to solve this problem. You should too.

The MMBU Challenge tests whether models can actually recognize, localize, and understand what is in biomedical images, not just arrive at the right answer.

🏆 3 tracks
💻 $100K+ in compute and prizes
📅 Oct 1 to Dec 31
⏰ Registration closes today

[AlphaSignal @AlphaSignalAI]

The article examines what belongs in AGENTS.md, design documents, and tests.

Full breakdown ↓
https://t.co/TkPph8xd2h

[AlphaSignal @AlphaSignalAI]

An agent repeats a mistake, so you add a rule.

Months later, nobody remembers why it exists.

“Why Does CLAUDE.md Keep Growing?” calls this “catastrophic remembering.”

The study tracks 247,694 instruction lifetimes across 1,867 repositories.

Files like CLAUDE.md more than triple over their lifetime.
The cause is lost reasoning, not changing requirements.

Without the reason, deleting anything feels risky.
The paper tests attaching a comment to each rule.

Those comments record:

> The failure behind the instruction
> The hypothesis behind the fix
> Evidence from previous attempts

The author built tests where the ideal prompt was known.
Uncommented prompts ended 211% longer than that ideal.

Informative comments cut the excess to 1.4%.
The agent followed constraints equally well both ways.

In an interview with AlphaSignal, author Kushal Chakrabarti (@kushalc) recommends a cache approach.

“The cache should have triggers and pointers, not summaries.”

[AlphaSignal @AlphaSignalAI]

Full breakdown: https://t.co/2JefTBps3o

[AlphaSignal @AlphaSignalAI]

A support ticket comes back as billing, sales, or technical. The return is that label, with a probability on it.

You pass the state, the questions, and the allowed answers. That exchange is one step.

TypeSafe's Jev does that in 70 to 500 ms. Input is $0.042 per million tokens, and output tokens are not metered.

> 40x to 200x faster on the comparison table, for System One shaped queries
> 444.6x cheaper only as the workflow-eval high end

A negative balance stays in code. The model is for the call you cannot separate with a rule.

> Route the ticket
> Score the urgency
> Pick the next tool

The vocabulary is the list you typed. A fraud ticket that only offers billing, sales, and technical comes back as one of those three. A spam score of 0.52 is not a 0.998

> Laya, 421 million parameters, about 33 ms for one question on a Tesla T4
> That 33 ms is a local pass, not the hosted 70 to 500 ms
> CLM, up to 9x faster zero-shot when actions repeat or the list is long
> Fine-tuned CLM, 81.6% of 38 DeepSWE tasks and 87.6% of 30 Terminal-Bench 2.1 tasks
> Jev on that judge test, 71.1% and 83.1%, under the one-sample line

Which real ticket has no label in your list?
Full breakdown↓↓

[AlphaSignal @AlphaSignalAI]

Your audio can score well on DNSMOS and still hurt your AI model.

That’s the problem we’re digging into with @Voices at our upcoming virtual workshop.

We’ll look at the audio quality issues traditional scoring can miss, why reverb matters for TTS, voice cloning & ASR, and how to tell when audio is actually training-ready.

Plus, you’ll hear the difference live.

Join us → https://t.co/MHdIWh2isw

[AlphaSignal @AlphaSignalAI]

Full breakdown: https://t.co/MbbwJ1LC3a

[AlphaSignal @AlphaSignalAI]

People keep lining Qwen3.8-27B up with Opus 4.6, the coding frontier from a few months ago.

So, can that class of model sit on a consumer Mac?

Yes, but you still need to build the stack around it:
> an engine that quotes peak RAM
> a harness with a stop
> and tests you own

If you already use Claude Code or Codex, this is a private loop beside that bill.

Qwen3.8-27B fits a Mac as a 17.6 GB file. But MTPLX still peaks at 23.6 GB once the draft-ahead head is on.

A speed taken on a 128 GB Mac with thinking turned off is a different machine, so leave it out of a 24 GB average.

Two ways this goes wrong:

> You stop at the download, and 24 GB of RAM only loads it tight. A 4-bit copy there is still perfect for most tasks. 32 GB is the first real agent. 48 GB can hold a /goal that a machine can check. 16 GB is not this model.

> Or you stop at the first hour. Ollama is a fine look, but after that you still pick an engine, a harness, and a stop.

Local tokens have no invoice, so a runaway does not get expensive, and you still own the merge. Tests are the stop.

We put the pulls, the install, the skill files, and the git-config check on an interactive HTML tutorial.

Which file to pick if you already live in MLX, and what to refuse to train this week, are a longer argument.

Full breakdown below

[AlphaSignal @AlphaSignalAI]

Your AI agents are generating a ton of operational data.

Prompts. Tool calls. Latency. Tokens. Errors. Retries. Cost. Outcomes.

The hard part is turning all of that into something you can actually monitor.

In our upcoming workshop with @TigerDatabase, we’ll show how to build an agent monitoring system on Postgres + time-series data, then scale it for teams running hundreds or thousands of agents.

If AI agents are moving into production, this is the infrastructure layer you’ll need.

Join us → https://t.co/3tBSWo6dL8

[AlphaSignal @AlphaSignalAI]

The article applies these checks to an illustrative agent dashboard.

Full breakdown ↓↓https://t.co/3gbqFexpnl

[AlphaSignal @AlphaSignalAI]

METR’s vibe-coded dashboard exposed an agent through an authentication flaw.

An attacker prompted it to reveal its API key.

Over three weeks, the attacker consumed roughly $600K in credits. The provider supplied them free; METR reported no financial loss.

A successful login doesn’t test what happens without one.

Even an HTTP 401 can hide an unauthorized action. If the backend queues work before authentication, a worker can still execute it.

Check the response, queue, and task database together.

Then trace which credentials the worker can read. A separate inference service can hold the provider key and enforce access and usage limits.

Before you deploy:

> Test that rejected requests create no jobs
> Check access across different user accounts
> Keep provider keys outside code-execution workers
> Restrict worker permissions
> Test credential revocation and recovery

[AlphaSignal @AlphaSignalAI]

Grok 4.7 just landed on @mercor's APEX leaderboards:

> #5 on APEX-SWE at 53.6%
> #13 on APEX-Agents at 54.6%

The notable part is the bracket, which sits in the same cost and latency tier as Gemini 3.7 Flash and GPT-5.6 Luna while leading on SWE.

APEX updates with every major model release, so you can see how the newest models stack up, compare token costs, and explore results from third-party benchmarks like BrowseComp and Terminal-Bench.

See the full leaderboard at https://t.co/7sAoPQZDh7

*partnerpost

[Mercor @mercor]

Grok 4.7 is now live on the APEX leaderboards.

APEX-SWE: 53.6% Pass@1 (#5)
APEX-Agents: 54.6% Pass@1 (#13)

Compared to other models with similar cost and latency profiles, it’s a strong model for agentic coding tasks.

Congrats to the @spacexai team.

[AlphaSignal @AlphaSignalAI]

A good DNSMOS score doesn’t always mean your audio is ready for AI training.

Join AlphaSignal + @voices for a technical deep dive into what DNSMOS misses, including the “grey zone” where reverb can quietly degrade TTS, voice cloning & ASR models.

We’ll cover:

→ Where DNSMOS falls short
→ How to spot problematic audio
→ What training-ready audio actually sounds like
→ Live audio comparisons

Register → https://t.co/MHdIWh2Qi4

[AlphaSignal @AlphaSignalAI]

Full breakdown by @bendee983: https://t.co/HOOAk4FByZ

Taste answer: the lean 4-tool setup held the frontier on both benchmarks, so start from the lean setup and add weight only when a failure or a needed feature earns it.

The full check, the 9-of-12 vendor-default split, and when to pay more stay in the full breakdown.

[AlphaSignal @AlphaSignalAI]

Same model bills 5x more in a different harness

HarnessTax tested 21 model-harness combos across 60 tasks and cost moved far more than success did

> $1.33 against $0.67 per attempt
> 97.8% versus 96.7% success rate
> 9 of 12 beat vendor default

Claude Code and Pi averaged 15 turns, yet the heavy setup carried more than 10 times the initial context because longer instructions and larger tool definitions ride along on every call.

Study co-author @melissapan told us many harness choices run on tribal knowledge and word of mouth, so it is very unclear which setup wins on accuracy or cost for your workload.

We read this as a buying test and the lean setup still holds the cost-success frontier on four tools, read, write, edit and bash.

> When does the heavier harness earn its higher price?
> Which failures hide when two setups post the same score?
> What would your own tasks show if you priced model and harness as one system?

Answers below ↓↓

[Melissa Pan @melissapan]

Does your Claude model really need Claude Code…? 🤔

We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge:

1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost
2️⃣A simple harness can be competitive
3️⃣The native harness isn’t always the best.

Millions of people are using coding agents, but the impact of harness choice remains unclear.

(1/n) More details in the thread. 🧵

[AlphaSignal @AlphaSignalAI]

@edgarpavlovsky you just need AlphaSignal my guy. It's freeeeeeee and loved by 335k plus engineers