Interesting read from Anthropic, with several parallels to our work on agent memory evaluation. Last week, we shared an approach for turning any existing benchmark into a memory evaluation by exploiting dependencies between tasks.
https://t.co/gDQ8BNHsfK
Memory is what allows Claude to learn from experience. In a new Engineering Blog post, we share how “dreaming” helps Claude organize what it’s learned, correct its mistakes, and find what it needs more easily the next time. https://t.co/uUEN4bNkr9
How can we turn existing benchmarks into memory evaluations with a small reframing?
Benchmark items are normally treated as independent. But tasks may share information, such as source documents, sections, facts, or intermediate calculations.
We can measure this overlap at different levels, depending on the type of memory we want to evaluate. For each task i, let Eᵢ represent its information at the selected level. We define directional coverage as: D(i → j) = |Eᵢ ∩ Eⱼ| / |Eⱼ|. This measures how much of task j’s information also appears in task i.
We then construct a directed acyclic graph (DAG), where each node represents a task. Edges point from earlier to later tasks in the evaluation order, weighted by their directional coverage. Direction represents evaluation order, not causality. Edge weight represents the proportion of information needed by the later task that was already encountered in the earlier task.
Every task remains independently solvable from the original sources. Memory is not required, but the DAG identifies where information acquired during one task could be reused in a later task. We can then compare runs with persistent memory against runs that reset memory between tasks to measure whether the agent benefits from information acquired during earlier tasks. This gives us a simple way to evaluate the benefit of memory using existing benchmarks, without changing their original tasks, labels, or scoring.
4/4 Interestingly, the cost improvement also held on out-of-distribution questions, without a performance drop.
One possibility is that reorganization discovers a better page granularity, rather than simply fitting the wiki to past questions. That is only one interpretation of the result.
1/4 Can a wiki reorganize itself around the questions it receives to reduce inference-time token usage?
We’ve been exploring this idea recently: treat wiki structure itself as an optimization problem.
3/4 Testing with Claude Opus 5.0, cost on unseen future questions declined over successive rounds of reorganization.
We also saw a small performance gain as a byproduct, although accuracy was not part of the objective.
@JMRothberg (whose team sequenced the first individual human genome) once spoke to my Yale BME class. I nearly joined the many friends he recruited across New Haven
What he built at @4Catalyzer feels like a modern Bell Labs for biotech: bring exceptional scientists and engineers together, give them funding to attack impossibly ambitious problems, and turn the results into companies
That model produced @ButterflyNetInc (ultrasound-on-a-chip), @Hyperfine (portable MRI), IsoPlexis (single-cell proteomics), Detect (at-home molecular diagnostics), Quantum-Si (next-generation protein sequencing), and more
But the most impressive output may be the people who left to build again:
Eric Kauderer-Abrams → Head of @AnthropicAI Life Sciences
@dalvabaird → CEO Crownlands (human neuroscience neolab)
Matt de Jonge → CEO Fortell (AI hearing aids)
Gregory Charvat → CEO TeraDAR (advanced sensing)
Jonathan Naccache → CEO Brink Therapeutics (programmable gene integration)
Gino Korolev →CEO Fertu (digital-health engagement)
@nicodjimenez → founder Mathpix (AI document conversion)
@marcosrortiz4 → Cofounder stealth (AI x bio)
Kam Firouzi → CEO Althea Health (AI voice agents for healthcare)
@ayodaiken → CTO Stealth AI native financial services
Michael Bellos + Faisal Razi Ahmad→ cofounders Arcsecond Labs (optical engineering)
and the flywheel is still spinning: Butterfly’s ultrasound chips power @midjourney Medical’s full-body scanner
The best founders don’t just create companies. They create ecosystems that produce the next generation of founders
(and, remarkably, most of this group is barely on Twitter and maintains a low profile. they’ve been too busy building.... maybe that says everything)
No one has cracked the nut on how to properly run evals on agents, the lack of standardization and transparency combined with the extreme variance, arguably infinite number of variables, and cost makes this a difficult problem. Evaluations have major economic, scientific and political implications, and right now it’s a complete shit show
We Built the Agents Faster Than We Built the Rulers
I spent a decade doing research in theoretical astrophysics and cosmology. Compared to that field, AI benchmarks have been more vibes than science...(rant incoming)
Best dinner with some of the GREATS across AI x life science here in NYC
While Alfredo may object, NY (plus Boston + London) is cooking👨🍳
SVPs who've run drug development at big pharma, ML researchers, founders building vertically integrated manufacturing infra - all at one table
In typical fashion, we went around the table for intros - everyone shared their name and their Roman Empire (h/t Collin) / latest obsession
A real privilege to host and get these minds in one room! More NY events to come (runs, science talks, poker nights, and dinners)🥳stay tuned
@MattStergiou@tzeevis@mingyu_ma@constable_todd@mpcadosch@gruver_nate@MichaelRetchin@patrickmineault@Collin_Spencer_@max_strome@marcosrortiz4@Killianhmmm@ZahraKhwaja
A single run is often a poor estimate of a long-horizon agentic pipeline’s typical behavior. Small perturbations can compound into very different outcomes, but running enough trials to characterize the outcome distribution is expensive. We need sample-efficient evaluation methods with finite-sample guarantees on estimation error.
1/ What can a benchmark from 1976 teach us about today's AI evals? The story of Whetstone. The first synthetic benchmark in computer science. A short thread.
7/ Modern AI benchmarks often score the final answer. Whetstone reminds us to also inspect the path. Only then can we tell whether we measured the capability we cared about, or an unintended shortcut through the evaluation.