Elias Wehbe

[Elias Wehbe @eliaswehbe]

@paulg We can’t look at these graphs without looking at operating margins when trying to tell the story of how fast AI startups grow compared to other software companies. They might grow super fast but never capture enough value.

[Elias Wehbe @eliaswehbe]

@garrytan We’re trying to hire more engineers specifically because we can do so much more with AI. Why does everyone seem to believe we’ve saturated the markets we can serve with software? They’re about to blow up!

[Elias Wehbe @eliaswehbe]

This is so good

[Andrej Karpathy @karpathy]

Excited to release new repo: nanochat!
(it's among the most unhinged I've written).

Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI.

It weighs ~8,000 lines of imo quite clean code to:

- Train the tokenizer using a new Rust implementation
- Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics
- Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use.
- SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval)
- RL the model optionally on GSM8K with "GRPO"
- Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI.
- Write a single markdown report card, summarizing and gamifying the whole thing.

Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.

My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved.

Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.

[Andrej Karpathy @karpathy]

Excited to release new repo: nanochat!
(it's among the most unhinged I've written).

Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI.

It weighs ~8,000 lines of imo quite clean code to:

- Train the tokenizer using a new Rust implementation
- Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics
- Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use.
- SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval)
- RL the model optionally on GSM8K with "GRPO"
- Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI.
- Write a single markdown report card, summarizing and gamifying the whole thing.

Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.

My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved.

Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.

[Elias Wehbe @eliaswehbe]

@s_ibarraran @thinkymachines Why are there accuracy peaks at specific steps across all models? What causes that behavior?

[Elias Wehbe @eliaswehbe]

@Jack_W_Lindsey This graph suggests it's harder to "steer away" from an answer than to "force" an answer. Is steering basically overwriting features with features for "correct" in the part of the model with the wrong answer?

[Elias Wehbe @eliaswehbe]

@KielWeb @MarioNawfal @grok @grok there’s an error in the original message can you find it?

[Elias Wehbe @eliaswehbe]

@amasad Algos don’t pick them up

[Elias Wehbe @eliaswehbe]

@chamath So you mean consulting?

[Jay Yang @Jayyanginspires]

This is the way

[Elias Wehbe @eliaswehbe]

@tobi Extension pack is equally good and a little spooky

[Elias Wehbe @eliaswehbe]

@Walgreens please fix this Arabic.

[Elias Wehbe @eliaswehbe]

We keep on talking about how much faster AI adoption is than other technologies but we never mention that the distribution channels and cost of distribution of AI is infinitely lower than the internment (we needed service providers and computers) and integrated circuits. I wonder if there’s a way to normalize this growth. Still the most consequential technology of our lifetimes.

[Elias Wehbe @eliaswehbe]

@andrewchen I’m personally interested in seeing vibe coding come to the analytics world where we’ll be doing vibe analytics vs writing tons of repetitive code

[Elias Wehbe @eliaswehbe]

@ChrisJBakke Should be a new reality TV show

[Elias Wehbe @eliaswehbe]

@paulg ie Most AI solutions right now trying to build on existing workflows

[Elias Wehbe @eliaswehbe]

@gregisenberg Working on 2. Closed yearly deal with EY (they have tons of data analysts who need this). It’s more cursos agent than copilot

[Elias Wehbe @eliaswehbe]

@KonstantHacker Is the X-axis performance? Seems like an inverted chart.

[Elias Wehbe @eliaswehbe]

@NathanielMc 3350. Interested?

[Elias Wehbe @eliaswehbe]

A great song and a video to match. http://t.co/CdtBUNaNVh