//sagar

Hacker, Programmer, Computer Geek, AI and Blockchain engineer

[Andrej Karpathy @karpathy]

Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project.

This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.:

- It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work.
- It found that the Value Embeddings really like regularization and I wasn't applying any (oops).
- It found that my banded attention was too conservative (i forgot to tune it).
- It found that AdamW betas were all messed up.
- It tuned the weight decay schedule.
- It tuned the network initialization.

This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism.
https://t.co/WAz8aIztKT

All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges.

And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.

[Chintan Turakhia @chintanturakhia]

giving out 100 @baseapp invites today.

we love thoughtful and helpful feedback, so keep it coming.

reply and RT if you want one.

[//sagar @ultimateg33k]

@chintanturakhia @baseapp would love to get a invite @chintanturakhia

[Vendetta 📀 @TheVendettaGame]

Welcome to Black Haven.

[Vendetta 📀 @TheVendettaGame]

Vendetta hits the ground in Nigeria! 🇳🇬

Our first in-person event was a success at the Sui Gaming Meetup in Ado City, hosted by the incredible @darakrypt!

Big love to @SuiNetworkNG for diving into the Sui gaming ecosystem and building with us.

This is just the beginning!

[//sagar @ultimateg33k]

@TheVendettaGame 🔥🔥

[Vendetta 📀 @TheVendettaGame]

The Vendetta DVD NFTs are about to drop - DVDs you'll need to play the game. But before you claim the turfs, it's time to take over X. Add a 📀 next to your name, reply to this post & show your support.

The gangsters are assembling. The takeover starts NOW!

[//sagar @ultimateg33k]

@antinertia Agent

[OpenAI @OpenAI]

Agent Tools for Developers

https://t.co/giS4K1yNh9

[Nik McFly @nikmcfly69]

🤯 BREAKING: Manus AI created its own open-source alternative. In 25 min, it built a complete AI agent system from scratch!

ANUS (Autonomous Networked Utility System)—@eugeneshilow's brilliant idea

Why pay $20K for invites when it can be free?

#Manus #ManusAI #Cursor

[//sagar @ultimateg33k]

Bridging Chains Privately with ZKPs

#ZeFiDay #Aleo

@Venture23xyz

[//sagar @ultimateg33k]

https://t.co/9lxJ4cOfrn [IRL]
https://t.co/acqyfzZnSN [Virtual]

#ZeFiDay

[DeepSeek @deepseek_ai]

🚀 Day 0: Warming up for #OpenSourceWeek!

We're a tiny team @deepseek_ai exploring AGI. Starting next week, we'll be open-sourcing 5 repos, sharing our small but sincere progress with full transparency.

These humble building blocks in our online service have been documented, deployed and battle-tested in production.

As part of the open-source community, we believe that every line shared becomes collective momentum that accelerates the journey.

Daily unlocks are coming soon. No ivory towers - just pure garage-energy and community-driven innovation.

[GangstaVerse 📀 @GangstaVerse]

Gangsters from @GangstaVerse are making their way over to #Sui - ready to take over the streets in a whole new way! 🚀🔥

Stay tuned for the next big move.

#OnChainGaming

[//sagar @ultimateg33k]

An excellent book by Google DeepMind for scaling language models using TPUs.
https://t.co/NnOFHRcPL6

[//sagar @ultimateg33k]

@SuiFamOfficial
Great people great event.. Go sui...

[Varun Navani @VarunNavani]

In 2017, Tesla faced an impossible challenge:

Teaching cars to see and understand the world like humans.

Then they hired a 29-year-old genius who solved it.

But what he did next shocked everyone in Silicon Valley.

Here's the untold story of Tesla's AI mastermind:

[//sagar @ultimateg33k]

@NVIDIAGeForce #GeForceRTX50

[//sagar @ultimateg33k]

Exploring the latest in #VirtualProtocol: As digital landscapes evolve, understanding virtual protocols is key to seamless connectivity and innovation. Dive into the trends shaping the future of virtual interactions. What are your thoughts? 💡 #TechTrends #Innovation

[Kiosk Indexer @KioskIndexer]

Calling all #Sui Developers!

Kiosk Indexer is live in testnet with mainnet launch just around the corner!

Join our Telegram group to connect with the community and stay updated.

#SuiDevs #KioskCommunity #BuildOnSui

https://t.co/VvgJjhmOBT