Justin Johnson

F500 AI exec who still ships. Author of Builder-Leader. The builds, the numbers, what broke. The Blog →

[Justin Johnson @builderleader]

https://t.co/OgoaJG4HxM

[Justin Johnson @builderleader]

Model choice has stopped being the bottleneck for agents; how they are orchestrated and how their actions are proven now matter more.

WorkflowArena shows the strongest model can be the worst coordinator, and Microsoft backdoored an agent by editing fewer than 10 neurons, while new tools for signed audit trails and file hashing go after that same gap.

With FP8 KV caches in llama.cpp and Apple reportedly packaging local models for the Mac, agents on clinical data can now run inside the firewall, and their provenance layer is the part still missing.

Link in comment.

[Justin Johnson @builderleader]

https://t.co/m3Wx2soTol

[Justin Johnson @builderleader]

Fable 5.2 makes frontier-quality work cheaper per call, which moves the risk into the layers around the model: compacted context, agent memory and the evaluation that certifies results.

Each of those layers failed quietly in new evidence.

Compaction erased numbers and audit pointers in its first pass, task outcomes hid whether memory was used at all, and choosing the test step alone inflated survival concordance by as much as most published gains.

A clinical agent pipeline that compacts context or reports a C-index needs type-aware retention and a locked evaluation protocol before a cheaper model goes into it.

Link in comment.

[Justin Johnson @builderleader]

https://t.co/hc6OlEXh1i

[Justin Johnson @builderleader]

Anthropic and Google both studied how scientists use their AI models, and the answer points at the lab bench.

Google surveyed scientists in the US and UK, and almost half said their main bottleneck had moved downstream in the past two years, into lab work and writing up. Anthropic's usage data leans the same way. People use AI where the work is text and code, and where they can check the answer at their desk.

I've been writing this post in different clothes all year. AI made the easy part of science cheap and left the slow part slow. In biology and chemistry, checking a result usually means running the experiment, and a cell line doesn't care how fast you wrote the protocol.

All three big labs are moving toward the bench. Anthropic runs a wet lab and has started running drug programs. Google has agents running liquid-handling robots. OpenAI rents a cloud lab and bought software to keep drug programs moving.

Checking is the product. Whoever owns the bench, the evals and the controls holds the value.

The scientist's job is moving from doing the work to judging it.

I think this is good for science. The labs will also learn what pharma already knows, that biology doesn't scale like code. Welcome to the industry.

Full piece on Run Data Run.

"The bench was never the cheap part."

[Justin Johnson @builderleader]

https://t.co/HMWQwMNQ47

[Justin Johnson @builderleader]

Frontier-scale open weights now fit on hardware a single team can buy: a 284B DeepSeek runs on 64GB of RAM, and a 2.8T Kimi runs on one Blackwell.

The constraint has moved to the agent layer, where compaction keeps rules but drops about 89% of numbers and 99% of exact commands in its first pass, and where Goldman and Apple are fencing and rationing coding agents.

Any agent pipeline that touches clinical data needs lab values, doses and provenance pointers stored outside the compacted context, because the summary will not keep them.

Link in comment.

[Justin Johnson @builderleader]

https://t.co/2fdoczYTLR

[Justin Johnson @builderleader]

Real-time avatars, video generation and scheduled web search all got cheaper today, while the failures clustered in agent memory, permissions and verification.

A CIKM paper shows Claude Code's /compact keeps only 10% of safety rules after five rounds, and a separate scan found Claude Code permission files leaking credentials in 1,122 public GitHub files.

The work worth funding is anything that checks agent output, from type-aware context retention to Lean-verified proofs to audit-ready clinical validation.

Link in comment.

[Justin Johnson @builderleader]

https://t.co/iICIzQF7V2

[Justin Johnson @builderleader]

I benched EmbeddingGemma 2 against Qwen3-Embedding-0.6B on 249 research notes: 30 graded queries, two CPU threads, no GPU.

Top-1 recall was 29 of 30 against 26 of 30. It indexed 2.3x faster and answered queries 2.2x faster. Cut to 256 dimensions, it kept the same recall at a third of the index size.

Add a reranker at depth 15 and both hit 30 of 30. Both already had all 30 by recall@10.

Recall alone wouldn't move me off Qwen. 2.3x faster indexing on a weak box might.

[Justin Johnson @builderleader]

https://t.co/oa8PNWGJr4

[Justin Johnson @builderleader]

OpenAI released BioBasicsBench: 100 routine wet-lab tasks built on raw gels, plate photos, microscopy and sequencing reads. GPT-6 Astra earns 66.7% of the available credit, GPT-5.6 Sol 43.8%.

In one cloning task, Sol saw no colonies on the negative control plate and built its conclusions on that. Astra noticed the plate was labeled with the wrong antibiotic.

The best model still misses a third of the credit on work wet-lab scientists are expected to do reliably. OpenAI built the bench and its model tops it, so I want to see someone else run it.

[Justin Johnson @builderleader]

https://t.co/zoz7ZU7XND

[Justin Johnson @builderleader]

Agents are becoming the default worker in research and data systems, and what is scarce now is evidence that their work holds up at the point of use.

TRACER keeps discrimination across health systems yet diverges at the screening operating point, GPT-6 Astra reportedly underperforms selectively on safety tasks, and new infrastructure assumes the agent holding the keys may be the failure.

Per-site thresholds and agent write and delete scopes now matter more than model launches, though next week's Inkling 2 Small weights are worth testing.

Link in comment.

[Justin Johnson @builderleader]

https://t.co/7K1zSWs4LX

[Justin Johnson @builderleader]

Agents now carry a quoted unit price for hard work, about $1,270 per validated and patched Linux kernel bug (vendor benchmark), while large buyers cap budgets and consumers barely use agents.

Clinical AI shows the same shift from model to measurement, with cohort rather than encoder driving the external drop in glioma survival prediction.

The signals to watch are independent reproduction of the kernel run and whether validation budgets move from encoders to multi-site cohorts and longitudinal biomarkers.

Link in comment.

[Justin Johnson @builderleader]

https://t.co/2wUzOfNdBc

[Justin Johnson @builderleader]

Three independent results today say the same thing: the biology-plus-LLM systems being benchmarked are not actually reading biology, and the models doing the reading are getting smaller and cheaper.

BioReason scores 0.848 with shuffled DNA against 0.847 intact, while frontier models fail BioSimGym and BioResearchBench outright, and Revolut cut inference costs 90% by self-hosting.

Audit any genomics model you depend on for input reliance before trusting its outputs.

Link in comment.