Anders Hafreager

@Cognite. Physics lover, startup enthusiast and addicted to programming.

[Anders Hafreager @andershaf]

@knutarnold Det ble vel godkjent i Norge i fjor også? Altså at Tesla har en bil som kan brukes til å teste i Norge?

[Anders Hafreager @andershaf]

@curryman90 @Tesla @elonmusk @grok No European contries. I am waiting for it in Norway.

[Anders Hafreager @andershaf]

@elonmusk I have someone else's chat history in my grok. https://t.co/ItNqW3X9aq is for some reason appearing as one of my chats. Security problem? Conversation id "e46f2c39-75c5-4385-b6e0-248d71ae2700"

[Andrej Karpathy @karpathy]

Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project.

This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.:

- It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work.
- It found that the Value Embeddings really like regularization and I wasn't applying any (oops).
- It found that my banded attention was too conservative (i forgot to tune it).
- It found that AdamW betas were all messed up.
- It tuned the weight decay schedule.
- It tuned the network initialization.

This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism.
https://t.co/WAz8aIztKT

All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges.

And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.

[Anders Hafreager @andershaf]

@teslaeurope When can I see this in Norway?

[Anders Hafreager @andershaf]

@elonmusk How about Europe?

[Anders Hafreager @andershaf]

Yes! I created this https://t.co/faYBiqTaVf with ChatGPT 3.5 to show a friend that he could make and deploy games without knowing JavaScript himself.

[OpenAI @OpenAI]

https://t.co/4tVqeVFebt

[Anders Hafreager @andershaf]

Funny that Starlink explains that they will block X in Brazil, on the platform X.

[Starlink @Starlink]

To our customers in Brazil (who may not be able to read this as a result of X being blocked by @alexandre):

The Starlink team is doing everything possible to keep you connected.

Following last week’s order from @alexandre that froze Starlink’s finances and prevents Starlink from conducting financial transactions in Brazil, we immediately initiated legal proceedings in the Brazilian Supreme Court explaining the gross illegality of this order and asking the Court to unfreeze our assets. Regardless of the illegal treatment of Starlink in freezing of our assets, we are complying with the order to block access to X in Brazil.

We continue to pursue all legal avenues, as are others who agree that @alexandre’s recent orders violate the Brazilian constitution.

[Nathaniel @TheNastyNat]

Hey @elonmusk why didn't you tell us that Tesla sells convertibles now? Because the roof of our brand new model Y fell off on the highway

[John Carmack @ID_AA_Carmack]

You should never do a cross-fade in a stereoscopic production, because your eyes can't converge on the blended intermediate values.

[Anders Hafreager @andershaf]

@Meteorologene @yrnyheter Yr-appen har brukt 40% batteri i background activities siste 24 timer på min iPhone X (bruker også Apple Watch-ikonet for å vise temperatur). Vært en uheldig oppdatering?

[Anders Hafreager @andershaf]

I got 7.25 of 10 points on this @CppQuiz https://t.co/aZFi8hVWuS ! Can you match that? #cpp #cplusplus #lucky

[Pain Point @PainPoint]

Visualization of Modems Connecting

[Greg Egan @gregeganSF]

No individual colour making up this wave travels faster than the grid lines, but the sum of all components appears to outrace the grid. This is how “group velocities” can exceed the speed of light without carrying information.
https://t.co/mBhvDFW6IL

[Anders Hafreager @andershaf]

@jorgenem @Meteorologene @yrnyheter :D Det er kaldt der nå. Det fant jeg jo ut fortere enn noe annet!

[Anders Hafreager @andershaf]

@Meteorologene @jorgenem Jeg skal til Roma i påsken og lurte på hvordan været blir. Fant ikke Roma og skal ikke til Romarheim. Med tanke på at den viser Roma når jeg søker på Rome er det rom for forbedring?

[Svenn-Arne Dragly @dragly]

Working with @andershaf to make it possible to select individual particles in Atomify. In the process, we found and fixed a bug in #Qt3D where shaders would be purged from the cache too early. @andershaf is fixing another bug we found for deferred rendering on macOS now :) #qtdev

[Anders Hafreager @andershaf]

Tenker på dette: https://t.co/FI84Yk7xa7. Ser nå at det står "WiFi Tale kan ikke brukes i utlandet". Så det svarte jo på spm mitt :) https://t.co/q31JLVGLlR

[Telenor Norge @TelenorNorge]

@andershaf Så sant du er koblet til WiFi og kan ringe over WiFi nettet så vil det ikke bruke data, da bruker du jo WiFi på samme måten som her hjemme. Men forklar gjerne mer spesifikt hva du tenkte på med WiFi calling :)

[Anders Hafreager @andershaf]

@TelenorNorge Jeg er i USA og dere har uanstendig høye prise for samtaler... :( Ringer jeg gratis med Wi-Fi Calling?

[Anders Hafreager @andershaf]

@1Password I have an old account without secret key, but I'd like to have a new account. Is it possible to somehow transfer the account? Or is it not due to new pricing plans?