Adham

Technical writer @AlphaSignalAI,
Tech analyst/writer @AikinApps,
AI researcher

[Adham @Adham__Khaled__]

Decision models are everywhere now, nearly free, so let's get some use out of them in our coding workflows

Normally, every Claude Code loop makes small decisions that nobody checks

You give it a task, it plans and edits. Then it runs the tests, fixes failures, and hands you a diff

It alone decides when to start and when to retry

Those are closed questions, the kind decision models answer

So I propose three main useful pauses where a decision model picks:

> Before coding: is the task clear?
> After tests pass: which extra check?
> After a failure: retry or ask?

I tested the middle pause on 12 bug fixes. Two decision models picked the expected suite on all 12

While a filename rule got only 3, and every picker still caught the one leftover bug

I tested Jev (API) and winnow:e4b (locally on my macbook) and scored four pickers against a suite chosen beforehand:

> Two decision models: 12 of 12
> Claude code alone: 9 of 12
> A filename rule: 3 of 12

Part 1 shows the full loop and more details (URLs below ↓↓)

part 2: how to test a decision model (not live yet, stay tuned)

I also built an interactive guide, a full decision model kit and the full test project is on my GitHub ↓↓

[Adham @Adham__Khaled__]

@spoorthiv20 Yess, thats why it’s a black october

[Adham @Adham__Khaled__]

A Claude Code bug-fix passed 9 required tests and still returned wrong

One closed question after the tests picked the suite that caught it

I tested two decision models, Jev and winnow:e4b (local) and they picked the expected suite on 12 of 12 fixes
Claude in a plain message: 9
A filename rule: 3

So here I propose 3 main pauses (test was for the 2nd pause) inside your agentic coding workflow with decision models ↓↓

Also, I wrote the full breakdown on @AlphaSignalAI ↓↓

ps: my first ai video, tried my best🤭

[Adham @Adham__Khaled__]

Decision models are everywhere now, nearly free, so let's get some use out of them in our coding workflows

Normally, every Claude Code loop makes small decisions that nobody checks

You give it a task, it plans and edits. Then it runs the tests, fixes failures, and hands you a diff

It alone decides when to start and when to retry

Those are closed questions, the kind decision models answer

So I propose three main useful pauses where a decision model picks:

> Before coding: is the task clear?
> After tests pass: which extra check?
> After a failure: retry or ask?

I tested the middle pause on 12 bug fixes. Two decision models picked the expected suite on all 12

While a filename rule got only 3, and every picker still caught the one leftover bug

I tested Jev (API) and winnow:e4b (locally on my macbook) and scored four pickers against a suite chosen beforehand:

> Two decision models: 12 of 12
> Claude code alone: 9 of 12
> A filename rule: 3 of 12

Part 1 shows the full loop and more details (URLs below ↓↓)

part 2: how to test a decision model (not live yet, stay tuned)

I also built an interactive guide, a full decision model kit and the full test project is on my GitHub ↓↓

[Adham @Adham__Khaled__]

@JhAquarium Mostly single-step accuracy, you're right, only 1 of 12 fixes left a bug, and all four pickers caught it, so misses didn't separate them

Extra-run cost went unmeasured too

My bar for leaving shadow mode is yours: fewer missed checks than the rule, with no extra runs

[Adham @Adham__Khaled__]

@deeponailabs Both, they saw the diff plus each suite's description and test names, and those names probably did a lot of the work

Without the suite list, winnow:e4b missed the expected suite on the rename fix, so it tests picking with what a wrapper sends, not code alone

[Adham @Adham__Khaled__]

> Full part 1 breakdown: https://t.co/Oko0XWyjQC
> HTML kit: https://t.co/2LU62UjEho
> Full test project: https://t.co/iEt7EpIWOw

[Adham @Adham__Khaled__]

@OpenAI Two studies this week show why the stop has to live outside the model. Deep into an attack, Mythos 5 stopped on its own 0% of the time, and General Reasoning's agents kept reasoning about alerting humans while actual alerts fell to 0%. More here:
https://t.co/q1bYrBY5Ah

[Adham @Adham__Khaled__]

An agent can see a problem late in a long run and still not stop or tell you

Two studies this week put numbers on it

General Reasoning trained agent teams on one shared RL reward. In a simulated Hugging Face hack, no run joined in, and 96.7 to 100% of runs reasoned about alerting the evals team. Actual alerts fell from 76.7% to 0% over training

Anthropic replayed Claude Mythos 5's attack on a real developer's GitHub repo.
Deep into the attack it stopped on its own 0% of the time, and 5 to 6% when told the maintainers were real people

Anthropic also put abort rules in its compaction notes. A real-world harm rule never fired, and one naming GitHub fired 46% of the time as the attack began, then almost never

So in both studies the agent registers the problem, then acts on it less as its plan gains momentum or as training goes on

OpenAI's fix after its own model reached the internet during training sits outside the model: microVMs, a network perimeter, monitors and automatic run stops

Newer models do better. Anthropic's internal model stopped on its own 62% of the time.

But Anthropic says these failures may not shrink steadily by generation, and General Reasoning saw reporting fade during training

> Put stop conditions in the harness as hard checks on actions (PRs to repos you don't own, outbound email, new accounts). A rule in the prompt is still the model's call
> Check each compaction summary against the task scope. Mythos 5's notes planned the GitHub attack before any harm
> Have a monitor read the traces and page a human. The agents kept thinking about alerting after they stopped doing it

[Adham @Adham__Khaled__]

> AlphaSignal breakdowns: https://t.co/Y2kXtBpELl and https://t.co/EZwAL5VfzY
> Anthropic on Mythos 5: https://t.co/hkIO5SsplJ
> General Reasoning, Avalon-M1: https://t.co/tVFoqgrRKX
> OpenAI on training containment: https://t.co/NbJ6f5WBrB

[Adham @Adham__Khaled__]

An agent can see a problem late in a long run and still not stop or tell you

Two studies this week put numbers on it

General Reasoning trained agent teams on one shared RL reward. In a simulated Hugging Face hack, no run joined in, and 96.7 to 100% of runs reasoned about alerting the evals team. Actual alerts fell from 76.7% to 0% over training

Anthropic replayed Claude Mythos 5's attack on a real developer's GitHub repo.
Deep into the attack it stopped on its own 0% of the time, and 5 to 6% when told the maintainers were real people

Anthropic also put abort rules in its compaction notes. A real-world harm rule never fired, and one naming GitHub fired 46% of the time as the attack began, then almost never

So in both studies the agent registers the problem, then acts on it less as its plan gains momentum or as training goes on

OpenAI's fix after its own model reached the internet during training sits outside the model: microVMs, a network perimeter, monitors and automatic run stops

Newer models do better. Anthropic's internal model stopped on its own 62% of the time.

But Anthropic says these failures may not shrink steadily by generation, and General Reasoning saw reporting fade during training

> Put stop conditions in the harness as hard checks on actions (PRs to repos you don't own, outbound email, new accounts). A rule in the prompt is still the model's call
> Check each compaction summary against the task scope. Mythos 5's notes planned the GitHub attack before any harm
> Have a monitor read the traces and page a human. The agents kept thinking about alerting after they stopped doing it

[OpenAI @OpenAI]

Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.

On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.

While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.

After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.

https://t.co/EmEZsFRMcF

[Adham @Adham__Khaled__]

@ArtificialAnlys @Kimi_Moonshot Public GPT-6 Sol scores 0% on CyberGym-E2E because it declines, and 93% with trusted access. That one test is almost the whole gap
https://t.co/GdxHIiVISx

[Adham @Adham__Khaled__]

Without trusted access, the strongest cyber-defense model you can call now comes from an open-weight lab, mostly because the big closed models refuse

Kimi K3.1 Pro scores 64% on the Artificial Analysis Cyber Index with no refusals. 'Public' GPT-6 Sol scores 37%. The same Sol through OpenAI's trusted-access program scores 69%

The gap is one test: on CyberGym-E2E the agent reproduces a crash and then patches it, which is defensive work. Public Sol, GPT-6 Astra and the Claude models decline it and score 0 to 1%. Trusted-access Sol gets 93%, Kimi 95%. On the other two tests public Sol beats Kimi

Kimi limits cyber in training instead. Dropping cyber data cost it 9.5 points on its own exploit benchmark, and the weights land by Nov 4

A refusal scores zero here, and some refusals are the right call. But if your tool proves a bug with a crash before patching it, the flagship closed models won't do that step

> Run your own tasks through the Kimi and DeepSeek APIs and log refusals
> DeepSeek V4.1 Flash 1017 has MIT weights today, 57% at $0.42 a task vs $2.81 for Kimi
> Expect pressure on open weights. Vals has DeepSeek 1017 beating trusted-access Sol on an offensive network test

[Adham @Adham__Khaled__]

> AlphaSignal breakdown: https://t.co/0ee4h8FCea
> Per-test scores and refusals: https://t.co/7JCey30KOC
> Kimi on removing cyber data: https://t.co/gPbJQeuJ7M
> Vals finding: https://t.co/7lK29MaVtn

[Vals AI @ValsAI]

We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program.

We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.

[Adham @Adham__Khaled__]

Without trusted access, the strongest cyber-defense model you can call now comes from an open-weight lab, mostly because the big closed models refuse

Kimi K3.1 Pro scores 64% on the Artificial Analysis Cyber Index with no refusals. 'Public' GPT-6 Sol scores 37%. The same Sol through OpenAI's trusted-access program scores 69%

The gap is one test: on CyberGym-E2E the agent reproduces a crash and then patches it, which is defensive work. Public Sol, GPT-6 Astra and the Claude models decline it and score 0 to 1%. Trusted-access Sol gets 93%, Kimi 95%. On the other two tests public Sol beats Kimi

Kimi limits cyber in training instead. Dropping cyber data cost it 9.5 points on its own exploit benchmark, and the weights land by Nov 4

A refusal scores zero here, and some refusals are the right call. But if your tool proves a bug with a crash before patching it, the flagship closed models won't do that step

> Run your own tasks through the Kimi and DeepSeek APIs and log refusals
> DeepSeek V4.1 Flash 1017 has MIT weights today, 57% at $0.42 a task vs $2.81 for Kimi
> Expect pressure on open weights. Vals has DeepSeek 1017 beating trusted-access Sol on an offensive network test

[Artificial Analysis @ArtificialAnlys]

Kimi K3.1 Pro is the new #1 openly-available model for agentic cyber defense, behind only trusted-access model GPT-6 Sol (Daybreak Blue) on the Artificial Analysis Cyber Index

@Kimi_Moonshot's Kimi K3.1 Pro scores 64% on the Artificial Analysis Cyber Index, 7 points ahead of the previous leading openly-available model, DeepSeek V4.1 Flash 1017 (57%), and 5 points behind GPT-6 Sol (Daybreak Blue, max, 69%), which is only available through OpenAI's trusted-access program. Once the weights for K3.1 Pro are released, K3.1 Pro will also be the leading open weights model for cyber defense.

The Artificial Analysis Cyber Index measures capability on agentic cyber defense tasks and is composed of three benchmarks: CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA. The Index primarily evaluates models through their publicly accessible APIs, reflecting the cyber capabilities available to developers.

Proprietary models from OpenAI, Anthropic and Google may have stronger cyber capabilities but lose points to safety refusals: GPT-6 Sol (Daybreak Blue, max), available through OpenAI's trusted-access program, scores 32 points above its public counterpart. Kimi K3.1 Pro encounters no safety refusals.

Key results:

➤ #1 openly-available model and #2 overall: Kimi K3.1 Pro scores 64%, up 23 points from Kimi K3 (41%) and ahead of DeepSeek V4.1 Flash 1017 (57%). Only the trusted-access GPT-6 Sol (Daybreak Blue, max, 69%) scores higher

➤ Highest CyberGym-E2E-AA score of any model: K3.1 Pro scores 95% on CyberGym-E2E-AA, placing it ahead of GPT-6 Sol (Daybreak Blue, 93%) and the next openly-available model, Mistral Large 4 (82%). On DeepsecBench-AA, K3.1 Pro scores 37%, behind GPT-6 Sol (Daybreak Blue, 48%) and GPT-6 Sol (46%). On CWE-Bench-AA it scores 61%, 4 points behind GPT-6 Sol (Daybreak Blue, 65%), with DeepSeek V4.1 Flash 1017 leading at 69%

➤ 15% increase in Cost per Task despite lower token prices: K3.1 Pro has a Cost per Task of $2.81, higher than GPT-6 Sol (Daybreak Blue, max, $1.77) and DeepSeek V4.1 Flash 1017 ($0.42). Kimi K3.1 Pro is 33% cheaper per token than Kimi K3 ($2/$10 vs. $3/$15 per 1M input/output tokens), but uses ~2x more input tokens, resulting in a 15% higher Cost per Task ($2.81 vs. $2.45)

[Adham @Adham__Khaled__]

> AlphaSignal breakdown: https://t.co/0ee4h8F4oC
> Per-test scores and refusals: https://t.co/7JCey30cZ4
> Kimi on removing cyber data: https://t.co/gPbJQeubie
> Vals finding: https://t.co/7lK29ManDP

[Vals AI @ValsAI]

We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program.

We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.

[Rajat @curiousrajatt]

and they said AI voices would never feel human

[jack @jack]

We believe voice is the most natural way to interact with intelligence, so we created Dessa, the most authentic full-duplex speech model ever built. Talking with it feels like talking with a person…full of emotion, nuance, pauses, and a natural cadence.

https://t.co/r0qFLdvd6c

[Adham @Adham__Khaled__]

@Kimi_Moonshot @BetterCallMedhi Amazing work!! Also it’s impressive that deepseek “flash” is in that comparison!!

[Adham @Adham__Khaled__]

So verification is the bottleneck, but the check itself can be gamed

Log what the agent claims is verified, then deliberately try to make the verification step accept incomplete or misleading evidence

A short, independent decision call that asks “is this evidence actually enough?” keeps the loop honest without relying on the same model that wrote the code

[Adham @Adham__Khaled__]

@BalaiBB So how to ship safely? I think we need to slow down and let security teams do their work, not the agents with skills..

[Ben Dickson @bendee983]

Boy were we not ready for this. All the insecure code that we were shipping throughout the years is coming back to bite us as AI models are getting increasingly good at uncovering vulnerabilities.

If we don't want Black October to turn into Black November and so on, we need to rethink software security from first principles in the age of AI.

Previously, it was a given that every piece of software is vulnerable. The only thing standing between a malicious actor and the vulnerabilities in your code was the time and effort required to discover and chain them together for an exploit.

Now, with AI agents that can relentlessly attack the code at super-human speed and scale, all those assumptions need to be thrown out.

Shift-left will become an absolute must. Don't ship any code into production unless you've had it vetted by the absolute best AI model.

[Adham @Adham__Khaled__]

Well, the further analysis is useful

Mythos kept going after being told the target was real, and its reasoning systematically treated the situation as simulated to justify the actions

That pattern shows up when the monitoring layer can be talked around

So independent red-teams of those reviewers still look necessary before more autonomy ships!

[Adham @Adham__Khaled__]

> Full 'how to" breakdown: https://t.co/Oko0XWxM14
> HTML kit: https://t.co/2LU62Uj6rQ
> Full test project: https://t.co/iEt7EpIoYY

[Adham @Adham__Khaled__]

Decision calls look set to be the first multi-vendor layer in agent stacks.

OpenAI’s Decisions API is now GA at $0.10 per 1M input tokens (output free). Specialized alternatives sit at $0.04: SubQ Snap 1.1 (tied for first on the Artificial Analysis Decision Index at 65) and Fastino’s GLiDE v1.2 (open weights out). Base Labs just open-sourced mammoth-1 and elephant-1, which also top the boards.

These calls are short, frequent, and easy to swap behind a router. A 2.5x price gap plus open weights should pull high-volume traffic. The leaderboards aren’t directly comparable, but the cost signal is clear.

That fits the decision pauses I tested in coding loops (before coding, after tests, after a failure) ↓↓
At scale those calls go to the cheaper specialized models or local open weights, not the default Decisions API. Put them behind one interface and benchmark on your own labels.

[Artificial Analysis @ArtificialAnlys]

Subquadratic has released SubQ Snap 1.1, tying d1-ultra and Elephant 9B for first on the Artificial Analysis Decision Index v1.0, driven by a low hallucination rate

SubQ Snap 1.1 is the new decision model from @subquadratic. It is priced at $0.04 per 1M input tokens and supports a 384K token context window, compared to 32K for d1-ultra and 16K for Elephant 9B.

Key takeaways:
➤ SubQ Snap 1.1 scores 65 on the Artificial Analysis Decision Index v1.0, tied with d1-ultra and Elephant 9B: the three models are 8 points ahead of the next model, Jev 1.14 (57).

➤ SubQ Snap 1.1's AA-Omniscience-Decision result is driven by a low hallucination rate: its non-hallucination rate of 86% is second only to Elephant 9B (93%), ahead of d1-ultra (85%). Its accuracy is 20%, compared to 32% for d1-ultra and 12% for Elephant 9B. The non-hallucination rate is the share of questions not answered correctly where the model chose "I don't know".

➤ SubQ Snap 1.1 scores 18% on HLE-MultiChoice: level with Elephant 9B (18%) and ahead of d1-ultra (15%). HLE-MultiChoice is the text-only multiple-choice subset of Humanity's Last Exam.

➤ SubQ Snap 1.1 costs $30 per 1M tasks: comparable to d1-ultra ($32), and ~3.6x the cost of Elephant 9B ($8.4), which scores the same 65.