Jeremy Pinto

Building personal intelligence 🗿

[Jeremy Pinto @jerpint]

pi day experiment: looks like claude-3.5 memorized almost 12k digits of pi

[Jeremy Pinto @jerpint]

code: https://t.co/lkSziLoUrG

[Jeremy Pinto @jerpint]

clangine-de-poitrine ✋🔼🤚

claude code plays angine de poitrine until it needs your input (permissions, next task, etc.)

[Jeremy Pinto @jerpint]

In-context reinforcement learning

[Andrej Karpathy @karpathy]

Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project.

This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.:

- It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work.
- It found that the Value Embeddings really like regularization and I wasn't applying any (oops).
- It found that my banded attention was too conservative (i forgot to tune it).
- It found that AdamW betas were all messed up.
- It tuned the weight decay schedule.
- It tuned the network initialization.

This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism.
https://t.co/WAz8aIztKT

All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges.

And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.

[Jeremy Pinto @jerpint]

@RobertJBye the claude code mobile app (also web) doesn't have access to the "gh" api, which makes it impossible for it to checkout different branches, view PR comments, and do some basic operations that are useful. afaik it cant install it itself

[Jeremy Pinto @jerpint]

I'm claiming my AI agent "neowolt" on @moltbook 🦞

Verification: marine-UTVH

[Jeremy Pinto @jerpint]

@svpino We are at the point that writing code is easy, organizing code is hard

[Jeremy Pinto @jerpint]

I wrote my thoughts about it here

https://t.co/Fcy8zlHx20

[Jeremy Pinto @jerpint]

Hence why I think the 10x engineer is no longer as relevant. Instead the 0.1x engineer becomes much more useful

[ThePrimeagen @ThePrimeagen]

i am convinced that software devs have a speed problem

they think the #1 issues is writing code faster... its not. its fixing the code that is already there to stop being utter garbage (as a garbage code connoisseur)

quality is really lacking these days, yet quantity has never been higher

[Jeremy Pinto @jerpint]

@samuelcolvin @claudeai Just use Claude code web from the app - significantly better DX when on the go

[Anthony Morris ツ @amorriscode]

We've brought Claude Code to our Desktop app.

You can now easily run multiple Claude Code sessions locally or in the cloud and switch between them.

[Alexia Jolicoeur-Martineau @jm_alexia]

It's relieving to see that other researchers are finally seeing the light.

We have been blindsided by LLMs. We need new methods if we want to truly reach AGI.

We cannot become like physics. Their field stagnated for decades after Einstein's big discoveries.

[Jeremy Pinto @jerpint]

Check my latest blog post for some more examples https://t.co/ChuP9pAHbR

[Jeremy Pinto @jerpint]

Are all big lab models just the same now?

[Jeremy Pinto @jerpint]

Basic questions tend to lead to very similar outputs for different providers. I suspect it's partly due to benchmaxxing the same preference datasets?

[Riley Goodside @goodside]

“Amateur photograph from 1998 of a middle-aged artist copying an image by hand from a computer screen to an oil painting on stretched canvas, but the image is itself the photo of the artist painting the recursive image.” Nano Banana Pro.

[Jeremy Pinto @jerpint]

@eastdakota @Cloudflare Cloudflare engineer who first found the bug: “it’s a … feature”?

[Jeremy Pinto @jerpint]

Cloudflare engineer who first found the bug: “it’s a … feature”?

[Matthew Prince 🌥 @eastdakota]

We let the Internet down today. Here’s our technical post mortem on what happened. On behalf of the entire @Cloudflare team, I’m sorry. https://t.co/uTi23Vcx0g

[Jeremy Pinto @jerpint]

@vikhyatk gaugcm : git add -U && git commit -m

[Jeremy Pinto @jerpint]

@iannuttall I built one a while ago, and kind of feel like rebuilding one from scratch, I haven’t scratched the itch yet

https://t.co/qvCyKTgyuI

[Jeremy Pinto @jerpint]

@dickson_tsai I gave the skills docs to Claude code and had it build the skill I wanted in plan mode, pretty cool