Shane Conway

This sentence is false.

[Shane Conway @statalgo]

@JeffBezos Terrible idea. Ignores the primary information source of the price system.

[Shane Conway @statalgo]

@stevemagness Or this guy.

[Andrej Karpathy @karpathy]

Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project.

This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.:

- It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work.
- It found that the Value Embeddings really like regularization and I wasn't applying any (oops).
- It found that my banded attention was too conservative (i forgot to tune it).
- It found that AdamW betas were all messed up.
- It tuned the weight decay schedule.
- It tuned the network initialization.

This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism.
https://t.co/WAz8aIztKT

All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges.

And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.

[F. A. Hayek Quotes @FAHayekSays]

Hayek was asked to leave “a statement for the future generations.” His response is brilliant:

“Modern civilization which enables us to maintain 4 billion people was made possible by the institution of private property. It is only thanks to this institution that we achieved an extensive order far exceeding anybody’s knowledge.”

“If you destroy that moral basis, which consists in the recognition of private property, we will destroy the sources which nourish present-day mankind, and create a catastrophe of starvation beyond anything mankind has yet experienced.”

[Shane Conway @statalgo]

@RandPaul He did his PhD under Samuel Huntington and has always had a neo-Conservative proclivity.

[Tour de France™ @LeTour]

🔁 RT if you think 🇮🇪 Ben Healy is the Super-combative of the #TDF2025!

🔁 RT si vous pensez que 🇮🇪 Ben Healy est le Super-combatif du #TDF2025 !

[Shane Conway @statalgo]

@MebFaber Easy. @EconTalker with the always entertaining @mungowitz on capitalism. https://t.co/sIv61p5r7t

[Shane Conway @statalgo]

@DylanoA4 My absolute favorite book.

[Shane Conway @statalgo]

@stevemagness 1:53 in 8th grade, and impressive family performances too. Strong genes there.

[Shane Conway @statalgo]

@jgault13 Graham Blanks is destined for greatness. Incredible.

[Senator Rand Paul @SenRandPaul]

Tariffs are taxes, and the power to tax belongs to Congress. Our Founders were clear: tax policy should never rest in the hands of one person. Tariffs not only drive up costs for American families but also trample on the Constitution.

It’s time Congress reasserts its authority and restores the balance of power.
https://t.co/xkxZaKGWvz

[Dimitri Bertsekas @DBertsekas]

I am pleased to share the full set of videolectures, slides, textbook, and other supporting material of the 7th offering of my Reinforcement Learning class at ASU, which was completed two days ago; check https://t.co/HSeALvVHyM

[Rand Paul @RandPaul]

Congress Must Assert Itself on Tariffs | National Review https://t.co/pHSwTOvLcz

[Shane Conway @statalgo]

@ProfEmilyOster Very neat, thanks! Will definitely follow. Although probably not significant without a single business day (ie. no traffic on a Sunday at that time before the event).

[Shane Conway @statalgo]

6. Sean Carroll "The Big Picture: On the Origins of Life, Meaning, and the Universe Itself” https://t.co/OsQHlCdSmT Unifying framework for quantum physics and philosophy. Very thought provoking about scales of theories.

[Shane Conway @statalgo]

My favorite reads this year (not necessarily *from* this year). Top recommendations followed by honorable mentions. (1/16)

[Shane Conway @statalgo]

5. Koyama and Rubin “How the World Became Rich: The Historical Origins of Economic Growth” https://t.co/bEjWkR0m3z Re-read more closely. The best book on growth, and a model for academic writing. Must read.