ok it's official... AI-written unit tests and integration tests are empirically proven to be unhelpful. you should tell your agents to stop writing tests by themselves
on the deepswe eval set, banning sonnet 5.5 high from writing any tests actually resulted in slightly higher success rate (non stat-sig), with less time and token spent (stat-sig). this is pretty hard empirical evidence
across the tests that were written by the "tests allowed" baseline arm, 65% of them were unit tests, 35% were integration tests, neither bucket resulted in any improvement compared to not writing any tests at all
i also picked a random sample of 44 tasks subset where i completely disabled executing even existing tests - it also did not affect success rate at all
i spot checked many tests written in the baseline arm, and my intuition is that most tests are simply a repetition of the implementation
agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself
i suspect we can still extract some value from unit tests and integration tests if we describe them ourselves when we believe we can articulate our intent better through test cases than through requirements. but i have not proven this yet
also worth noting, during deepswe eval the agent wrote almost no e2e tests (only 17, compared to 3000+ unit/integration tests written). so this analysis does not prove nor disprove the value of e2e tests - i will do another eval specifically for that
Instead of watching an hour of Netflix, watch this 2 hour hour Stanford lecture will teach you more about how LLMs like ChatGPT and Claude are built than most people working at top AI companies learn in their entire careers.
Eleven v3 is out of alpha and ready for commercial use.
Since alpha, we've improved stability and accuracy:
- Stability: more reliable model and higher user preference scores
- Accuracy: 68% fewer errors on numbers, symbols, and technical notation
China just turned the night sky into a masterpiece of precision and intelligence. 🌌
What struck me most about this is how seamlessly innovation turns into art.
A drone show in Chongqing just broke the Guinness World Record with 11,787 synchronized drones, creating breathtaking 3D animations that looked closer to CGI than real life.
No human pilots.
No delays.
No crashes.
Every movement guided by AI and GPS, choreographed with perfect timing.
To me, this is far more than a light show.
It’s a glimpse into how technology, creativity, and coordination can merge to shape a new era of expression and innovation.
When intelligence takes flight, it doesn’t just illuminate the sky — it redefines what’s possible.
Could this be the moment where technology begins to turn the world itself into its stage?
#AI #Innovation #Technology #Drones #Automation #Creativity #China #Engineering #FutureOfWork #DigitalArt
Credits: longliveai
INTRODUCING MORPHEUS-1
The world’s first multi-modal generative ultrasonic transformer designed to induce and stabilize lucid dreams.
Available for beta users Spring 2024
@letsgetrealtal1 @pupsoapy @GregAbbott_TX Yea, I live in Texas now. From California. In California i lost power for > two weeks out of every year for the last 4 years due to intentional or accidental shutdowns. Texas, a couple of hours last year. If you think Texas is bad, try out PG&E for awhile.
I’m claiming 5 $BAT for linking my @Brave Rewards wallet with my new Gemini account. You can too! Learn more: https://t.co/p79OXBCKXh #BeBraveWithGemini#EarnBAT via @Gemini