Andy

[Andy @_aandyw]

@NotJustBeats1 😂

[Andy @_aandyw]

https://t.co/VqPQ4jlulr

[Andy @_aandyw]

https://t.co/ARLyRTonOQ

[Andy @_aandyw]

https://t.co/amP3c1f8n3

[Andy @_aandyw]

@aryagxr beautiful notes as always!

[Andy @_aandyw]

@NotJustBeats1 i love it

[Andy @_aandyw]

just implemented pagerank in spark today.

in retrospect ... it's not even that bad lol

[Andy @_aandyw]

Amazing overview of llm post-training

Can’t believe this stuff is free…

https://t.co/34kDImfIuA

[Andrej Karpathy @karpathy]

Excited to release new repo: nanochat!
(it's among the most unhinged I've written).

Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI.

It weighs ~8,000 lines of imo quite clean code to:

- Train the tokenizer using a new Rust implementation
- Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics
- Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use.
- SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval)
- RL the model optionally on GSM8K with "GRPO"
- Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI.
- Write a single markdown report card, summarizing and gamifying the whole thing.

Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.

My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved.

Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.

[Andy @_aandyw]

Good overview of current AI eval techniques:
https://t.co/4DeY9GCz1m

[Andy @_aandyw]

To Read:

Advice on ML Research
https://t.co/lfVovSUv5d

[Thinking Machines @thinkymachines]

Introducing Tinker: a flexible API for fine-tuning language models.

Write training loops in Python on your laptop; we'll run them on distributed GPUs.

Private beta starts today. We can't wait to see what researchers and developers build with cutting-edge open models!

https://t.co/tJsgxgBuWo

[Andy @_aandyw]

@aryagxr You should build that :D

[Andy @_aandyw]

@asusevski yeah, i guess what I'm trying to get at is that I think there is growth in stability.

like if you develop a good routine and habits I feel like that naturally propels you to higher heights.

but maybe both things can be true? e.g. have stable life + take more risks in career?

[Andy @_aandyw]

If stability = comfort and comfort = limited growth ... then does that mean there's no growth in stability?

Personally, I feel like this is wrong

[Andy @_aandyw]

Sierra blog post on Tau-Bench: https://t.co/wS2PyG79S8

[Andy @_aandyw]

GDPval reminds me of Tau-Bench in that they're both trying to eval knowledge work in the real world. However, seems like GDPval is more focused on one-shot (as a PoC) whereas Tau-Bench is more interactive multi-turn.

[OpenAI @OpenAI]

Today we’re introducing GDPval, a new evaluation that measures AI on real-world, economically valuable tasks.

Evals ground progress in evidence instead of speculation and help track how AI improves at the kind of work that matters most.
https://t.co/uKPPDldVNS

[Andy @_aandyw]

- Benchmark verification comes from evaluating whether or not the database state obtained at the end matches the desired state
- pass^k used to measure how many times out of k a model can solve the same situation

[Andy @_aandyw]

@jonxuxu @bephrem Excited to see more soon! Very cool stuff