Interesting read from Anthropic, with several parallels to our work on agent memory evaluation. Last week, we shared an approach for turning any existing benchmark into a memory evaluation by exploiting dependencies between tasks.
https://t.co/gDQ8BNHsfK
Reproducibly wrong is still wrong. Cool results on detecting AI research failures from a single trace, without an answer key, even when repeated runs agree
Can you catch an AI agent's wrong answer from one run, with no answer key, and use it to generate tasks that expose new failures? One JEV read of the trace can rank failures (even quiet failures where many runs agree) and guide hill-climbing. Code is open. https://t.co/RCHEkTZyCk
How can we turn existing benchmarks into memory evaluations with a small reframing?
Benchmark items are normally treated as independent. But tasks may share information, such as source documents, sections, facts, or intermediate calculations.
We can measure this overlap at different levels, depending on the type of memory we want to evaluate. For each task i, let Eᵢ represent its information at the selected level. We define directional coverage as: D(i → j) = |Eᵢ ∩ Eⱼ| / |Eⱼ|. This measures how much of task j’s information also appears in task i.
We then construct a directed acyclic graph (DAG), where each node represents a task. Edges point from earlier to later tasks in the evaluation order, weighted by their directional coverage. Direction represents evaluation order, not causality. Edge weight represents the proportion of information needed by the later task that was already encountered in the earlier task.
Every task remains independently solvable from the original sources. Memory is not required, but the DAG identifies where information acquired during one task could be reused in a later task. We can then compare runs with persistent memory against runs that reset memory between tasks to measure whether the agent benefits from information acquired during earlier tasks. This gives us a simple way to evaluate the benefit of memory using existing benchmarks, without changing their original tasks, labels, or scoring.
Only 2.5 years ago, AI labs would highlight their frontier LLM performance on grade school math benchmarks (GSM8K). I can't wait to see where we'll be in another 2.5 years.
So basically, today PHP got two critical bugs that let anyone run code on your server without even logging in, which is bad, and then the details leaked before the patch was even out, which is worse, and it works on WordPress and MediaWiki and Magento and Joomla and Laravel, and Censys says like 1.5 million servers are running a version with no fix, so the NSA was like "take your system offline," and the FBI was like "take your system offline," and CISA and the UK and CERT-EU were all like "take your system offline," and then the Prime Minister of Japan, the actual Prime Minister, posted at 5:55 in the morning telling 48.8 million people to take their system offline, so everyone took their system offline, like the Dutch cybersecurity agency took its own website offline, and the Dutch parliament went offline, and the Port of Rotterdam went offline, and Finland and France and Denmark started pulling government sites, and Wikipedia went down, and https://t.co/QlOXb3COq3 went down, and Asana and GOG and Something Awful went down, and the University of Vienna's webmail went down for some reason, and that's just PHP. Because GitLab also dropped a CVSS 10.0, which is the highest score they give out, for a bug that's been there since 2014, so that's every self-hosted GitLab for twelve years, and then LiteLLM 1.107.0 got hijacked and shipped a credential stealer and a remote access trojan in every single wheel and the Docker images, including latest, and you didn't even have to import it, just installing it was enough, and it's the second time this year, and then Railway paused every deployment on the entire platform because someone got into the environment variables, so now everyone has to rotate every secret they've ever stored there, and TiDB Cloud said someone was walking around inside with stolen credentials for basically a month and an extortion group says they took 10.6 terabytes, and Zeabur got breached for the second time in two months and is just shutting down, like the whole company, and Node.js patched remote code execution and a TLS bypass, and CISA says Plex is being actively exploited, and someone took over 300 Tor exit relays to steal SSH passwords, and SafePay ransomware has hit over 600 critical infrastructure orgs, and India's biggest bank filed a data loss incident, and Japan has been in an official state of emergency in cyberspace since October 9, which honestly explains the 5:55 AM post. Oh, and eMudhra, a certificate authority that's issued over 60 million certificates, whose entire job is proving websites are who they say they are, was supposed to be using private keys that are random numbers over a hundred digits long, and one of them was 4. Just 4. And the others were like 66 and 237. Anyway, it's Wednesday.
@gracexyzheng and @xinjin have been wonderful friends and collaborators of the team since the very beginning. very excited for this work to be out and looking forward to even more!
My dear friends, I’d like to show you my latest toy project
Introducing Handwired: turn your regular JavaScript code into a neural net!
- no training
- exact outputs vs the JS program
- fully differentiable
All runnable in a browser, without GPU
Really interesting results. A few questions: Was the graph built per matter or across the full firm corpus, without access to the evaluation questions? Which three models did you test, and did you use LAB’s standard agent harness and graders or a Harvey-specific setup? Do the reported costs include graph construction amortized across queries? And when documents are added or amended, does the graph update incrementally or require a rebuild?
We hosted our second event at our offices this time for a research colloquium (only the panel was all pharma!). Big thanks to Manny (COO at Lexeo therapeutics, Former Merck Drug Development Veteran); my good friend Jeremy Elser head of Life Sciences @PalantirTech; and Amar Drawid, Chief AI Officer @BioNTech_Group
it was a grounding talk sharing the realities of where the life science industry is today... some of the perspectives offered:
1) discovery is saturated (there isn't a bottleneck in assets, molecules to pursue etc); investments here are purely defensive (to increase shots on goal) but ROI is fuzzy. when asked which area is overfunded, all 3 said solutions in R&D. when asked where is underfunded, all agreed it was in CMC/manufacturing.
2) negative data (failed experiments, trials that didn't pan out) is essential, not throwaway. If you capture what was tried, why, and under what conditions, you stop people from repeating the same dead ends
3) AI's biggest win is closing the feedback loops across development. the holy grail is connecting the dots between the lab, manufacturing, trials, and regulatory work turns scattered learnings into better decisions
4) as AI moves into higher stakes work, we need evals that are actually grounded in scientific context and real outcomes, not generic benchmarks, so teams know when to trust an output, when to step in, and whether it's actually helping
we had over 100 people RSVP, and hope to open it up more at our next event in Sept👀
big thanks to @NeuroLuebbert for the support + @awscloud for sponsoring
1/4 Can a wiki reorganize itself around the questions it receives to reduce inference-time token usage?
We’ve been exploring this idea recently: treat wiki structure itself as an optimization problem.
RLMs can very much do true recursion at depth=1 :)
because you're in a REPL, you can write *real recursive programs* that *call subagents* and still not need "subsubagents" anywhere.
claude code can't do this -- you'd have to write a program in a file that execs "claude -p"
No one has cracked the nut on how to properly run evals on agents, the lack of standardization and transparency combined with the extreme variance, arguably infinite number of variables, and cost makes this a difficult problem. Evaluations have major economic, scientific and political implications, and right now it’s a complete shit show
GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games?
We investigated. The harness was not letting it remember what it had learned.
We found that enabling two API settings tripled our scores with 6x fewer output tokens.
We Built the Agents Faster Than We Built the Rulers
I spent a decade doing research in theoretical astrophysics and cosmology. Compared to that field, AI benchmarks have been more vibes than science...(rant incoming)