ok it's official... AI-written unit tests and integration tests are empirically proven to be unhelpful. you should tell your agents to stop writing tests by themselves
on the deepswe eval set, banning sonnet 5.5 high from writing any tests actually resulted in slightly higher success rate (non stat-sig), with less time and token spent (stat-sig). this is pretty hard empirical evidence
across the tests that were written by the "tests allowed" baseline arm, 65% of them were unit tests, 35% were integration tests, neither bucket resulted in any improvement compared to not writing any tests at all
i also picked a random sample of 44 tasks subset where i completely disabled executing even existing tests - it also did not affect success rate at all
i spot checked many tests written in the baseline arm, and my intuition is that most tests are simply a repetition of the implementation
agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself
i suspect we can still extract some value from unit tests and integration tests if we describe them ourselves when we believe we can articulate our intent better through test cases than through requirements. but i have not proven this yet
also worth noting, during deepswe eval the agent wrote almost no e2e tests (only 17, compared to 3000+ unit/integration tests written). so this analysis does not prove nor disprove the value of e2e tests - i will do another eval specifically for that
You can now train your own Decision model like Jev locally!
We increased Qwen3.5 0.8B’s aggregate accuracy from 20.7% to 74.3% across 3 decision benchmarks - on just 4GB VRAM.
Turn any LLM like Qwen3.8, Gemma 4 into decision models with our open-source Unsloth repo.
We fine-tuned with a Clef head using Unsloth and LoRA (r=64) for one epoch, increasing downstream accuracy from 30–37% to 78%.
GitHub: https://t.co/2kXqhhvLsb
Guide and Notebooks: https://t.co/qACsYehl1n
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?
I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev
• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions
AFAICT the shortest path to AI-based economic revolution
How have software engineering fundamentals changed with agentic coding? Here is our AI Engineering Skills map for software engineering fundamentals. https://t.co/cnRLj43DLs
The battle in AI is shaping up to be a battle for context.
Everything in AI is about making sure that agents are effective as possible. That effectiveness comes down to whether the agent has the right domain expertise, access to the right context and tools to work with, and are involved in workflow in a way that users can easily interact with, review its work, and incorporate it into the rest of the process.
As a consequence, the platforms that are able to capture and leverage the best and most context within their agents —and be able to pick the right models for the task- will be the place where agents do their best work. You can just look at coding agents, legal agents, or support agents as examples of what this looks like at scale.
This is why the applied AI layer has a lot more value than just being an LLM wrapper. The ability to organize the critical knowledge for the work being done, and maintain this knowledge in a governed way where only the right people and agents have access, and the ability to improve the context for agents more and more over time, is critical.
Over time, this layer will be able to route work between a variety of models, leveraging frontier intelligence for planning and orchestration and review, and a mix of lower cost models (open or closed) for the large volume of work between these tasks.
The applied layer is also in a good position to train and develop its own models as well that are purpose built for their domains. Never good to bet against the bitter lesson, but equally taking a near frontier base model and post training it for just one type of domain work can -in many cases- lower costs or deliver better performance for certain tasks.
Finally, this applied layer is also where most of the change management of the workflow will need to occur. This is why FDEs are so important at the applied layer, because this is the point where the customer needs to have specific business problem solved by a particular vendor. Whichever companies can solve that completely in an end-to-end fashion will have the greatest moats.
As each day goes on, we’re learning more about what the likely long term market dynamics will look like in AI.
🚨Anthropic just showed a 24-minute workshop on how to actually do prompts for Claude.
Taught by the people who built it.
Free. No registration. No paywall.
I've seen $300 courses that don't cover what they teach in the first 8 minutes.
Watch it and bookmark it now.
At our latest YC Paper Club, researchers and builders presented on self-play for LLMs, AI for biology, formal verification, and agentic coding in production.
Thank you to our presenters:
00:00 — Francois Chaubard (@FrancoisChauba1) | Introduction & Call for Presentations
05:47 — Yasa Baig (@BaigYasa) | A World Model of Protein Biology (https://t.co/Swn2qAObv1)
25:38 — Luke Bailey (@LukeBailey181) | Scaling Self-Play with Self-Guidance (https://t.co/i5xefX0BkM)
37:51 — Arnab Maiti | Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage (https://t.co/5ZCfkKdbAJ)
47:40 — Robert Joseph George (@Robertljg) | Lean and the New Era of Verified Intelligence (https://t.co/ARWkAXjOyq)
58:52 — Lukens Orthwein (@lukensort) | Founder AI Hacks: Programming is an RTS Game Now
1:16:07 — Closing Remarks