Prajwal A

Building AI systems, breaking a few along the way
Experiments, research, and half-working demos

[Prajwal A @LawJarp_A]

@TheZachMueller @TheAhmadOsman Quite interested to see where this goes

[Prajwal A @LawJarp_A]

@ariG23498 Lesssgooo

[vLLM @vllm_project]

🚀 Day-0 MTP support for Gemma4 now available at vLLM with ready-to-use docker image!

⚡️Enjoy up to 3x faster decoding performance to supercharge your development with zero quality degradation!

Check out the full vLLM recipes for Gemma 4 model series👇
https://t.co/IrCaaa6SIo

[Prajwal A @LawJarp_A]

@getpeid @MKBHD @verge I've been using the phone(1) since its release. It is time to upgrade 🔥. Loving the new design

[Prajwal A @LawJarp_A]

@asmah2107 Hits too close to home

[Prajwal A @LawJarp_A]

@vllm_project Fun stuff 🔥

[Prajwal A @LawJarp_A]

Turns out if you use your steam id, and an steam api key, it's very simple to let prospective connections know what you work on and what you play xD

[Prajwal A @LawJarp_A]

Caching support now out on vllm-omni(@vllm_project)! Nice that I had gotten the opportunity to add TeaCache support!
https://t.co/RsUIfOBfpA

[Prajwal A @LawJarp_A]

@vllm_project Also, a shout-out to the incredible community of developers there helping me out.

[Prajwal A @LawJarp_A]

Just shipped TeaCache to vLLM-Omni!

Grateful to contribute to @vllm_project's multimodal expansion!
🔗 https://t.co/J5i68Cbrs7

[Prajwal A @LawJarp_A]

The magic: cache transformer blocks when consecutive timesteps look similar. The challenge: convincing each model architecture to play nicely.
Result: up to 3.36x speedup!

[Prajwal A @LawJarp_A]

@msharmavikram Lesgoo 🫡

[Prajwal A @LawJarp_A]

My model that has been working fine all this while has decided that a bag of chips and candy bars are one and the same.

[Prajwal A @LawJarp_A]

@kareem_carr Came for the tweet, stayed for your profile banner xD

[Prajwal A @LawJarp_A]

@TheAhmadOsman Karpathy out here making ‘build GPT from scratch’ sound like a sourdough recipe

[Prajwal A @LawJarp_A]

Finally posting instead of just lurking.
Vision, ML, and whatever works after 2 a.m :)

[Andrej Karpathy @karpathy]

nanochat now has a primordial identity and can talk a bit about itself and its capabilities (e.g. it knows it's nanochat d32 that cost $800, that it was built by me, that it can't speak languages other than English too well and why, etc.).

This kind of customization is all done through synthetic data generation and I uploaded a new example script to demonstrate. It's a bit subtle but by default LLMs have no inherent personality or any understanding of their own capabilities because they are not animal-like entities. They don't know what they are or what they can or can't do or know or don't know. All of it has to be explicit bolted on. This is done by asking a bigger LLM cousin to generate synthetic conversations (you tell it what they should look like simply in words), and then mixing them into midtraining and/or SFT stage. The most important challenge is ensuring enough entropy/diversity in your generated data. If you don't do it well, LLMs will generate 1000 conversations that are all ay too similar, even with high temperature. My script shows a crappy example of how to add diversity - e.g. by creating lists of starting messages or topics, sampling from them explicitly, adding them as fewshot examples into prompts for "inspiration", etc.

I wanted to have some fun with it so nanochat now refers to me as King Andrej Karpathy (lol) just to illustrate that this is a giant blank canvas - you can infuse completely arbitrarily identity, knowledge or style into your LLM in this manner. I hope it's helpful and sparks fun ideas!

[Andrej Karpathy @karpathy]

Excited to release new repo: nanochat!
(it's among the most unhinged I've written).

Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI.

It weighs ~8,000 lines of imo quite clean code to:

- Train the tokenizer using a new Rust implementation
- Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics
- Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use.
- SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval)
- RL the model optionally on GSM8K with "GRPO"
- Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI.
- Write a single markdown report card, summarizing and gamifying the whole thing.

Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.

My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved.

Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.

[Prajwal A @LawJarp_A]

@iScienceLuvr Interested

[Prajwal A @LawJarp_A]

@marktenenholtz Misconception that models are the answer to everything.
It's very surprising what quality feature engineering and post processing can accomplish