@aliromman Arguably, specialists are the best equipped to get high value out of AI models. Half of the battle is knowing which questions to ask.
On the other hand, AI also makes it easier for generalists to become specialists themselves, with focused effort.
Update: I’ve been making progress on all the other kernels using fleets of agents, and discovered human-guided fleets are over 2x faster than Astra-guided fleets!
We spent ~$5k testing fleets of agents running experiments on H200s.
Interesting results from the Open Kernel Challenge:
1) Human-guided fleets now progress over 2x faster than Astra guided fleets, and Astra alone is now too slow to make progress. Early on, human guided Astra made most of the progress based on intuition alone.
As the basic memory and GPGPU tricks are exhausted, most of the new kernels breakthroughs come from exhaustive testing on GPUs for optimizations and reorganizations of data/code.
So early boosts came from human insight, but now it's mostly based on how much you can spend on GPU testing.
2) Astra sometimes makes obvious mistakes a human expert would catch, like not combining other kernels into the full decode kernels, and not submitting breakthroughs for incorrect rules.
3) The best design is having a human expert keep adding agents till they respond every 1-5 minutes, then guide them with simple quick answers. I now run 30.
4) A human-guided fleet running Gemini 3.8 flash outperforms Astra Ultra alone and the Astra-guided fleet.
5) We spent ~$4500 in Gemini 3.8 flash running the fleet, 30 agents in parallel, and ~$600 in subscriptions running Astra, so the fleet is very expensive to run.
6) Gemini Argon kept stopping for security reasons so 3.8 Flash did better.
Most of the time was previously spent waiting for experiments, so I added agents till I didn't have to wait.
I built a local queue and lock system for all my GPUs, so they constantly have 100% utilization running independent experiments from the fleet.
Sometimes the fleet still goes off the rails, so humans need to re-prompt them. Astra can resolve some of this and verify the fleet, but when Astra goes off the rails you need a human expert. This explains the slowdown when Astra guides the fleet.
This is strong evidence that human experts guiding fleets of agents is both important to reach out-of-distribution solutions and make progress faster(for now).
NEW: Building on zxv's initial breakthrough, I've pushed the κ through a rapid series of improvements in just ~4 hours:
0.0023 → 0.00247755 → 0.00268647 → 0.00296 → 0.00300123
A 30.5% increase.
https://t.co/sbRrWbS6d0
Astra managed a swarm of 6.1 Sol for this.
“While the act of creation is inherently satisfying, there’s a danger: one might find satisfaction in the labor itself and mistakenly believe that a high-quality piece of work has been created,” he said.
very insightful comment from miyamoto
Everyone is talking about agent swarms, usually in the context of bad stuff like hacking. But if you ever wanted to watch an agent swarm in action (this time working for good), the discussion board of the Integer Multiplication open research repo is really cool to watch
I am finding that ChatGPT Pro is completely unusable on turns that take >2 hours to finish.
Has anyone noticed that long-running ChatGPT Pro sessions break completely?
It's just bugs all the way down. So frustrating, because ChatGPT Pro is unparalleled when it works.
Formal verification will soon be an everyday technology. I think we can solve a large % of all software security outright. If we’re lucky, formal verification will also be powerful tool for controlling AI risk
@pvncher When does it make sense to use multiple subagents vs main thread?
When do you use multiple threads, and what role do those threads play (e.g., manager, researcher, etc).