Nice paper on optimizing agents and their graders together.
The key finding is that you want to update your LLM grader alongside the agent instead of keeping it fixed, because the agent's failure modes change as it improves.
DUET alternates between the two.
Each round picks training tasks, runs the solver, has the grader score and critique the trajectories, then a tool-using updater revises either the grader or the solver while the other stays fixed.
On the Finance Agent Benchmark, the solver reaches 96.5%, above GEPA (94.4%) and SkillOpt (94.0%). It also beats both when they optimize against the benchmark's own reference evaluator.
Grader agreement rises from 68.4% to 78.5% on GDPval. Training the grader with the solver frozen moves agreement only from 67.9% to 68.8%.
Paper: https://t.co/qmZBl4RHu5
Color consistency is a persistent friction point in production. Our interactive research team just returned from The 19th European Conference on Computer Vision (@eccvconf), where they presented new work focused on solving just this: ensuring colors match reliably across shots and AI-generated reference photos.
If you explained agriculture to folks in SF, you'd blow so many minds. It's... self-replicating, recursively self-improving nanotech for the physical world. And it's all open-source!
One instance can turn into hundreds of seeds, that turn into hundreds of instances. Next thing you know you have trillions of them. It's super-exponential
Congratulations to our CEO @JensenHuang on receiving the National Medal of Science for advancing GPU computing to power scientific breakthroughs.
And to fellow honorees Elon Musk, Lisa Su, Michael Dell, Satya Nadella, and Sergey Brin.
John Urschel is an incredible guy. Had the pleasure of co-presenting an award with him at the Breakthrough Prize a few years ago - wonderful person and an inspiration!
i wrote approximately 2000 tweets and articles (like Deep Learning is Hitting A Wall) about the importance of supplementing deep learning/ LLMs with neurosymbolic AI over a span of 25 years and the tech bros are circulating the one time i forget to explicitly mention this, completely ignoring the rest of my work.
good grief.
You guys are so worried about web-based GEO that you forget to take advantage of EEO: email engine optimization.
I'm draftin' side notes to AI left and right 😉
In the past few months, we had a lot of fun (and stress 😅) to produce 12 versions (+ many subversions) and 137 pages in our training run log book.
Turns out human-human collaboration is important to improving human-AI collaboration. 😊
The Adolescence of Technology: an essay on the risks posed by powerful AI to national security, economies and democracy—and how we can defend against them: https://t.co/0phIiJjrmz
Workera is being acquired by Pearson! It's been a privilege to support CEO @kiankatan and the whole Workera team through this journey. Back in 2019, Kian had the insight that rigorous skills measurements would be important. With recent advances in AI, this is now more true than ever. Workera's technologies for rigorously measuring people's skills in different tasks and job roles are helping many businesses understand where their employees are strong and where there are areas for development.
With Workera and Pearson combining forces, I'm delighted that under @omarabbosh and Kian's leadership. Workera now has the potential to support an even larger number of people and businesses. Thank you, Kian and the Workera team for having me serve as the company's chairman, for launching Workera out of https://t.co/zpIxRSuky4 and AI Fund, for your friendship over the years, and for your important work advancing assessments.
OpenAI just dropped solutions to a whole bunch of hard Math problems
The AI solved more problems than what humans could solve in a decade
Expect AI to ace all science problems very soon!
If the target market is real time robotics, the Jetson Thor system, with 128 GB of memory, but only 273 GB/s of bandwidth, seems over-provisioned with expensive memory. You want the model evaluating at tens of fps, so it can't use more than 10 GB of weights at most.
More memory could be used if the model is a wide mixture of experts, or a large strategic planning model is operating in small time slices over a longer period of time, and more memory always makes development life easier, but for cost optimized systems, you should be able to get by with much less.
Excited to announce what we’ve been working on this year - @AnthropicAI, an AI safety and research company. If you’d like to help us combine safety research with scaling ML models while thinking about societal impacts, check out our careers page https://t.co/TVHA0t7VLc