Derek was my former manager, VP of eng at both MSFT and Atlassian
one of the most technical, forward thinking, hands-on eng leaders I’ve had the pleasure to work with
great to see him sharing on X and definitely recommend as a high quality connection!
Learn to build evals, folks.
Highly recommended reading on using decision models for evals.
From our internal evals, System One models like Jev are a great fit for LLM Judges.
If you want to go beyond Hamel's great write-up, I've also put together a few resources on applying decision models to LLM judges.
(bookmark it)
1) Jev-as-a-Judge for Agent Evaluations - https://t.co/SkXGd4uA2J
2) Jev-as-a-Judge: Evaluating Agents with a Decision Model - https://t.co/SqGSzlMFut
Bonus paper I would recommend, with a few other ideas that combine System One and System Two models: Jev-as-a-Judge paper - https://t.co/8Oi6oBEt2A
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks:
Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better:
Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better:
Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better:
Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work!
In summary:
- As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding.
- Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
Meta just change the game!
They open-sourced Context Language Models. Instead of using LangChain, RAG, or any orchestration harness, the model natively manages its own context like a file.
And it beats every SOTA context method with 59% less compute.
Instead of forcing a model to swallow endless walls of text like a passive file cabinet, they gave language models something revolutionary:
The ability to natively manage their own context.
To treat context like a dynamic file.
To learn their own internal policies entirely through weights, with zero human-designed frameworks or rigid retrieval harnesses.
The results shatter current benchmarks.
Using online reinforcement learning, they applied CLMs to the Qwen3.5-9B model.
Performance on BrowseComp-Plus skyrocketed by 47.6%.
And the wildest part? It used 12% fewer FLOPs to do it.
They didn't stop there. They co-designed Suffix Cache Reuse for server-side inference, slashing compute costs by 35% compared to standard SGLang at matched performance.
For years, labs have spent billions trying to engineer smarter ways to stuff more tokens into an LLM's prompt window.
They were solving the wrong problem.
Ilya Sutskever said: learn these 30 papers and you know 90% of what matters in AI.
this repo rebuilt all 30 in pure NumPy, No PyTorch, No TensorFlow, Just notebooks you can run.
- RNN
- LSTM
- Transformer
- ResNet
- VAE
- AIXI.
Ilya’s reading list, now executable.
Every paper in Jupyter, Synthetic data included,
Zero DL frameworks.
- https://t.co/GEAzryNQxm
jev + opus 5.5... i simply can't comprehend why everyone isn't building this yet.
in my workflow, this cut costs and time by ~80%. i think it's one of the best ways to use it.
→ pick relevant project notes before loading the context
→ route suitable tasks to a faster worker
→ choose a recovery path when a tool fails
→ run focused checks before the full test suite
opus handles the hard reasoning. jev picks from options the harness prepares and validates.
i explain how to build the decision layer in the article below:
Karpathy's prediction about RL is coming true now!
He argued that a single reward number tells an agent very little about why its work was good or bad. For complex tasks, he proposed a review informed by knowledge of the task.
That problem shows up as soon as RL moves beyond answers that can be checked automatically.
A math solution is right or wrong. Code can run against tests.
But an agent's work may involve dozens of decisions, and a successful final answer can hide wasted steps or poor tool use.
Engineers can write rules to score those trajectories, but the rules need to anticipate the behavior they want to reward. When the agent's workflow changes, the scoring code often needs another pass.
RULER, implemented in OpenPipe ART, lets the engineer write the evaluation criteria in plain English.
An LLM reviews multiple trajectories against those criteria and ranks them. ART turns that comparison into rewards for GRPO training.
I used this setup to train a Qwen3 1.4B agent to play 2048. It saw the board and chose a direction at each step.
Rather than scoring its moves with a collection of rules I had coded, I described what the reviewer should look for and let RULER compare complete attempts.
The result is still a numerical reward, but the difference is how the reward is defined. I can revise the description when I find behavior the reviewer is missing, then inspect whether its judgments match what I intended.
The full implementation is available on GitHub.
ART GitHub repo: https://t.co/fsoLXDK4Zu
(don't forget to star it ⭐)
I also wrote a walkthrough of the RL concepts and this training run, covering RLHF, GRPO, and RULER, in the article below.
This is actually insane. This uses @typesafeai Jev model, as a plugin in Claude to review all the un-nesseasary tool calls, and it takes 1s to run!
Like, literally, 1 second to take my Claude session from nearly 1M to ... 86K tokens! 😮
Ask your claude to install it and be amazed
Use this prompt
```
Install, and configure :
https://t.co/yD6PncSCtH
```
Computer scientists discover the power of *closed loop* and *error correction*! I have advocated for many years by now that all intelligent systems would be closed-loop, not open end-to-end. This has been the very basic lesson from Cybernetics (control theory and systems theory), originated more than 80 years ago. This view has been prominently featured in our open-sourced new book about the principles of intelligence: https://t.co/leZlkURb7j
Google's team published a 9-page PDF on Harness Engineering. one formula replaced prompt engineering:
Agent = Model + Harness
the twist: same Claude Sonnet, same benchmark. change only the harness
the playbook, six steps:
step 1 -> add guides
AGENTS.md, rule files, constraint docs. every line is a past agent failure turned into a permanent fix
step 2 -> add sensors
linters, tests, validation scripts the agent runs on its own output before a human sees it
step 3 -> build the agentic loop
plan, execute, verify, fix. bounded retries, budget caps, escalation when stuck
step 4 -> externalize memory
the model forgets every session. the harness holds state, decisions and artifacts across all of them
step 5 -> enforce permissions
which tools, how many writes, what needs approval. safety lives in the harness, never in the model
step 6 -> wire observability
track every tool call, cost and retry. trip wires fire when behavior drifts
the result: your agent turns from a demo into infrastructure
every failure makes the system permanently better, not just the next conversation
that's also the line between an agent you show people and one a client pays you to leave running
one harness gets you a reliable loop. connect several and it becomes graph engineering, where one agent's failure turns into another agent's sensor
the PDF stops at the harness. the graph is the next layer
that breakdown, running on Kimi, is below ↓
Two Anthropic researchers made Karpathy's agent loop 1000x better and they laid it out in an 11 page PDF
The big change is that agentic systems get dramatically better once you connect them into a graph instead of running them as isolated loops
here's the playbook in 6 steps:
Step 1: Build a single loop. Generate, critique, revise. One self review cycle beats a smarter model with no feedback at all
Step 2: Add tools. Search, code execution, a database. An agent that only thinks without using tools is essentially hallucinating
Step 3: Run agents in parallel. Separate worktrees, same repo, different branches, no conflicts.
Step 4: Add a graph. Agents write their findings as typed nodes and edges instead of transcripts. Every claim keeps its source attached
Step 5: Ground the evaluator in that graph. It checks claims against actual edges, not vibes. "Triple not found" is a much stronger signal than "seems off"
Step 6: The graph persists across sessions. Agents stop rebuilding context from scratch every time
Karpathy's original setup ran one agent in one direction. This runs a thousand agents with shared memory. Same model, different architecture, different results
Read the PDF, paste it into your own Claude
Google Brain founder, Andrew Ng: “Prompting will be dead in 6 months. Loops and Graphs are what’s replacing it.”
In 1 hour at Stanford, he shows what the best engineers are building instead — and how you can start building it today.
Everything a $500 course sells you is in the first 10 minutes.
Free.
Watch it, then read the guide below on how to build a system that improves itself.
Research papers that every AI Engineer must read part 2
1. Reinforcement Learning with Human Feedback (RLHF)
2. Mixture of Experts (MoE)
3. FlashAttention
4. Rotary Position Embedding (RoPE)
5. KV Cache
6. Chain of Thought Prompting
7. ReAct
8. GraphRAG
9. A2A Protocols
10. CLIP: Contrastive Language–Image Pre-training
11. BLIP: Bootstrapping Language-Image Pre-training
The prognosis of liver disease depends on the extent of fibrosis, which is usually staged with the use of liver biopsy. The limitations of biopsy have led to the development of noninvasive tests, which Drs. Laurent Castera, Mary E. Rinella, and Emmanuel A. Tsochatzis review. Learn more: https://t.co/oZ93UDdEiA