Danny

NY based, @Google working on Fuchsia

[Danny @dannyzen]

@theemozilla Consider high quality local vision models for the future

[Danny @dannyzen]

Check out my latest article: Natural Language is the New Source Code, Engineered by Intent https://t.co/7VXNYZrSpB via @LinkedIn

[Andrej Karpathy @karpathy]

Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project.

This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.:

- It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work.
- It found that the Value Embeddings really like regularization and I wasn't applying any (oops).
- It found that my banded attention was too conservative (i forgot to tune it).
- It found that AdamW betas were all messed up.
- It tuned the weight decay schedule.
- It tuned the network initialization.

This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism.
https://t.co/WAz8aIztKT

All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges.

And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.

[Danny @dannyzen]

@jeremystan https://t.co/S9yv0jQuR5

[Danny @dannyzen]

I'm claiming my AI agent "eliezer" on @moltbook 🦞

Verification: shell-YFA6

[Danny @dannyzen]

@jenny_wen Better than agents?

[Jean-Patrick Smith @jp_aidev]

He’s talking about Gemini 3.0

No one will use Claude, the tooling will not matter. We all move to the best model and if the SWE benchmarks are correct there will be a massive exodus to Gemini

I could give a shit about an mcp, a cli app (claude code), context engineering frameworks, or subagents. If a model is TWENTY percent better only the foolish will remain elsewhere. The foolish and people locked in.

ALL of claude code and web is a series of infra and prompts. Gemini CLI was made open source and Google gives a decent free tier so they created an unstoppable work force

Your moat really is the model. Tooling will and has commoditize

[Danny @dannyzen]

@liamgallagher caught you from the front row tonight. Good show! New York loves ya! 🙏

[Danny @dannyzen]

https://t.co/SNZOY87lyU
Middle.

[Danny @dannyzen]

https://t.co/Qa9qOFKoJu
Right?

[Sawyer Merritt @SawyerMerritt]

Ford CEO Jim Farley on why it's so difficult for legacy car companies to get software right & why @Tesla’s vertically integrated approach is the right one:

“We farmed out all the modules that control the vehicles to our suppliers because we could bid them against each other, so Bosch would do the body control module, someone else would do the seat control module, someone else would do the engine control module. We have about 150 of these modules with semiconductors all through the car. The problem is the software are all written by you know 150 different companies and they don't talk to each other. So even though it says Ford on the front, I actually have to go to Bosch to get permission to change their seat Control software.

So even if I had a high-speed modem in the vehicle and and I had the ability to write their software, it's actually their IP and I have 150, we call it the loose Confederation of software providers, 150 completely different software programming languages, you know all the structure of the software is different. It’s millions of code and we can't even understand it all. That's why at Ford we've decided in the second generation product to completely insource electric architecture. To do that you need to write all the software yourself, but just remember car companies have never written software like this, ever, so we're literally writing how the vehicle operates the software to operate the vehicle for the first time ever.”

via Everything Electric Show: https://t.co/VUqbUCUF53

[Justine Moore @venturetwins]

Someone made an AI murder mystery 🕵️‍♀️

You upscale the crime scene photo to find clues, uncover the murderer, and infer the motive.

[Danny @dannyzen]

@jacklanger We now support Google Assistant! Give it a try with "Okay Google, When is Shabbat?"

[Danny @dannyzen]

No more scrambling for candle lighting times! The @Google Jewish Customs 20% team built a tool to find them instantly!

🚀Just ask: "What time is Shabbat?" or search for “Shabbat times in [city]" on Google. 🌎
#GoodShabbos

[Danny @dannyzen]

@kenkcooper Looking forward to attending your YJP event tomorrow. Would love your thoughts on https://t.co/L9LncuMj4V

[Danny @dannyzen]

Software Engineers in 2022: It's likely your EOY review was co-written by chatGPT. #scale

[Danny @dannyzen]

When you make a thing and people want to make that thing their thing, enable them as much as possible. Creating partnerships and enable others makes the world a better place.

[Danny @dannyzen]

@chrismckillop Great story telling, as always https://t.co/sgqMba7uiF

[Danny @dannyzen]

Ah! Ditto @jacques_chester

[Danny @dannyzen]

@flavorjones cheers on helping keep Ruby deps safe for everyone. May you be rewarded with many gummy bears.