dkinz

Curious

[vLLM @vllm_project]

🚀 DeepSeek-OCR — the new frontier of OCR from @deepseek_ai , exploring optical context compression for LLMs, is running blazingly fast on vLLM ⚡ (~2500 tokens/s on A100-40G) — powered by vllm==0.8.5 for day-0 model support.

🧠 Compresses visual contexts up to 20× while keeping 97% OCR accuracy at <10×.
📄 Outperforms GOT-OCR2.0 & MinerU2.0 on OmniDocBench using fewer vision tokens.

🤝 The vLLM team is working with DeepSeek to bring official DeepSeek-OCR support into the next vLLM release — making multimodal inference even faster and easier to scale.

🔗 https://t.co/rnBG9VUuMy
#vLLM #DeepSeek #OCR #LLM #VisionAI #DeepLearning

[Andrej Karpathy @karpathy]

Excited to release new repo: nanochat!
(it's among the most unhinged I've written).

Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI.

It weighs ~8,000 lines of imo quite clean code to:

- Train the tokenizer using a new Rust implementation
- Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics
- Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use.
- SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval)
- RL the model optionally on GSM8K with "GRPO"
- Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI.
- Write a single markdown report card, summarizing and gamifying the whole thing.

Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.

My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved.

Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.

[RoboHub🤖 @XRoboHub]

A new dexterous hand is here. DexcelRobotics, a startup founded by a former core member of Tencent Robotics X, has launched its first product, the Apex Hand. The company claims it's the first in the industry capable of operating a cell phone with a single hand.

The Apex Hand is a well-rounded performer, with a focus on real-world application. Here are its key specs:

► Degrees of Freedom: 21, which the founder notes is enough to replicate most human hand functions.
► Strength: A single-finger force of ~2.5kg and a vertical lifting capacity of ~30kg.
► Speed & Precision: A response time near human-level, with positioning accuracy of ≤0.1mm.
► Robustness: Can withstand unexpected impacts and maintain stability.
► Tactile: Features self-developed flexible electronic skin with a sub-millisecond communication delay and >1000Hz refresh rate.

The founder explains that while academic research often focuses on single-point breakthroughs, a commercial product needs hardware stability, data handling, and model capabilities. The company’s full-stack development experience and modular design are key to bringing this technology to a broader market, starting with semi-structured industrial environments before reaching homes.

[Andrej Karpathy @karpathy]

+1 for "context engineering" over "prompt engineering".

People associate prompts with short task descriptions you'd give an LLM in your day-to-day use. When in every industrial-strength LLM app, context engineering is the delicate art and science of filling the context window with just the right information for the next step. Science because doing this right involves task descriptions and explanations, few shot examples, RAG, related (possibly multimodal) data, tools, state and history, compacting... Too little or of the wrong form and the LLM doesn't have the right context for optimal performance. Too much or too irrelevant and the LLM costs might go up and performance might come down. Doing this well is highly non-trivial. And art because of the guiding intuition around LLM psychology of people spirits.

On top of context engineering itself, an LLM app has to:
- break up problems just right into control flows
- pack the context windows just right
- dispatch calls to LLMs of the right kind and capability
- handle generation-verification UIUX flows
- a lot more - guardrails, security, evals, parallelism, prefetching, ...

So context engineering is just one small piece of an emerging thick layer of non-trivial software that coordinates individual LLM calls (and a lot more) into full LLM apps. The term "ChatGPT wrapper" is tired and really, really wrong.

[Andrej Karpathy @karpathy]

New 3h31m video on YouTube:
"Deep Dive into LLMs like ChatGPT"

This is a general audience deep dive into the Large Language Model (LLM) AI technology that powers ChatGPT and related products. It is covers the full training stack of how the models are developed, along with mental models of how to think about their "psychology", and how to get the best use them in practical applications.

We cover all the major stages:
1. pretraining: data, tokenization, Transformer neural network I/O and internals, inference, GPT-2 training example, Llama 3.1 base inference examples
2. supervised finetuning: conversations data, "LLM Psychology": hallucinations, tool use, knowledge/working memory, knowledge of self, models need tokens to think, spelling, jagged intelligence
3. reinforcement learning: practice makes perfect, DeepSeek-R1, AlphaGo, RLHF.

I designed this video for the "general audience" track of my videos, which I believe are accessible to most people, even without technical background. It should give you an intuitive understanding of the full training pipeline of LLMs like ChatGPT, with many examples along the way, and maybe some ways of thinking around current capabilities, where we are, and what's coming.

(Also, I have one "Intro to LLMs" video already from ~year ago, but that is just a re-recording of a random talk, so I wanted to loop around and do a lot more comprehensive version of this topic. They can still be combined, as the talk goes a lot deeper into other topics, e.g. LLM OS and LLM Security)

Hope it's fun & useful!
https://t.co/75mXcUBI8L

[garreth @garrxth]

🚀 With Meta's recent paper replacing tokenization in LLMs with patches 🩹, I figured that it's a great time to revisit how tokenization has evolved over the years using everyone's favourite medium - memes!

Let's take a trip down memory lane!

[1/N]

[Ahmad Al-Dahle @Ahmad_Al_Dahle]

With today’s launch of our Llama 3.1 collection of models we’re making history with the largest and most capable open source AI model ever released. 128K context length, multilingual support, and new safety tools. Download 405B and our improved 8B & 70B here. https://t.co/F8cI1bUL8h

[Andrej Karpathy @karpathy]

Actually, really liked the Apple Intelligence announcement. It must be a very exciting time at Apple as they layer AI on top of the entire OS. A few of the major themes.

Step 1 Multimodal I/O. Enable text/audio/image/video capability, both read and write. These are the native human APIs, so to speak.
Step 2 Agentic. Allow all parts of the OS and apps to inter-operate via "function calling"; kernel process LLM that can schedule and coordinate work across them given user queries.
Step 3 Frictionless. Fully integrate these features in a highly frictionless, fast, "always on", and contextual way. No going around copy pasting information, prompt engineering, or etc. Adapt the UI accordingly.
Step 4 Initiative. Don't perform a task given a prompt, anticipate the prompt, suggest, initiate.
Step 5 Delegation hierarchy. Move as much intelligence as you can on device (Apple Silicon very helpful and well-suited), but allow optional dispatch of work to cloud.
Step 6 Modularity. Allow the OS to access and support an entire and growing ecosystem of LLMs (e.g. ChatGPT announcement).
Step 7 Privacy. <3

We're quickly heading into a world where you can open up your phone and just say stuff. It talks back and it knows you. And it just works. Super exciting and as a user, quite looking forward to it.

[Andrej Karpathy @karpathy]

📽️ New 4 hour (lol) video lecture on YouTube:
"Let’s reproduce GPT-2 (124M)"
https://t.co/QTUdu8b0qh

The video ended up so long because it is... comprehensive: we start with empty file and end up with a GPT-2 (124M) model:
- first we build the GPT-2 network
- then we optimize it to train very fast
- then we set up the training run optimization and hyperparameters by referencing GPT-2 and GPT-3 papers
- then we bring up model evaluation, and
- then cross our fingers and go to sleep.
In the morning we look through the results and enjoy amusing model generations. Our "overnight" run even gets very close to the GPT-3 (124M) model. This video builds on the Zero To Hero series and at times references previous videos. You could also see this video as building my nanoGPT repo, which by the end is about 90% similar.

Github. The associated GitHub repo contains the full commit history so you can step through all of the code changes in the video, step by step.
https://t.co/BOzkxQ8at2

Chapters.
On a high level Section 1 is building up the network, a lot of this might be review. Section 2 is making the training fast. Section 3 is setting up the run. Section 4 is the results. In more detail:
00:00:00 intro: Let’s reproduce GPT-2 (124M)
00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint
00:13:47 SECTION 1: implementing the GPT-2 nn.Module
00:28:08 loading the huggingface/GPT-2 parameters
00:31:00 implementing the forward pass to get logits
00:33:31 sampling init, prefix tokens, tokenization
00:37:02 sampling loop
00:41:47 sample, auto-detect the device
00:45:50 let’s train: data batches (B,T) → logits (B,T,C)
00:52:53 cross entropy loss
00:56:42 optimization loop: overfit a single batch
01:02:00 data loader lite
01:06:14 parameter sharing wte and lm_head
01:13:47 model initialization: std 0.02, residual init
01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms
01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms
01:39:38 float16, gradient scalers, bfloat16, 300ms
01:48:15 torch.compile, Python overhead, kernel fusion, 130ms
02:00:18 flash attention, 96ms
02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms
02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping
02:21:06 learning rate scheduler: warmup + cosine decay
02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms
02:34:09 gradient accumulation
02:46:52 distributed data parallel (DDP)
03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)
03:23:10 validation data split, validation loss, sampling revive
03:28:23 evaluation: HellaSwag, starting the run
03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro
03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA
03:59:39 summary, phew, build-nanogpt github repo

[Andrej Karpathy @karpathy]

Awesome and highly useful: FineWeb-Edu 📚👏
High quality LLM dataset filtering the original 15 trillion FineWeb tokens to 1.3 trillion of the highest (educational) quality, as judged by a Llama 3 70B. +A highly detailed paper.

Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. You'd think it's random articles but it's not, it's weird data dumps, ad spam and SEO, terabytes of stock ticker updates, etc. And then there are diamonds mixed in there, the challenge is pick them out.

Pretraining datasets may also turn out to be quite useful for finetuning, because when you finetune a model into a specific domain (as is very common), you slowly lose general capability. The model starts to slowly forget things outside of the target domain. But this is not only restricted to knowledge; You also lose more general "thinking" skills that the original data demanded, but your new domain might not exercise. i.e. in addition to the broad knowledge fading, those computational circuits also slowly degrade. So there are likely creative ways to blend the pretraining and finetuning stages.

[Google AI @GoogleAI]

Infinite Wonderland is a Google Lab Session where four artists endlessly reimagine Alice’s Adventures in Wonderland using StyleDrop, a fine-tuning technique running on Imagen 2, to generate infinite images in their own, unique styles.

Check it out→ https://t.co/vGef1rUDDV

[Ashpreet Bedi @ashpreetbedi]

Building the LLM OS by @karpathy with gpt-4o

The speed + quality is 🔥🤯

code: https://t.co/jKX9zEFNik

[Google DeepMind @GoogleDeepMind]

Our next-generation AI model Gemini 1.5 Pro is now available in public preview on @GoogleCloud’s #VertexAI platform.

Its long-context window is already helping businesses analyze large amounts of data, build AI-powered customer service agents &amp; more. → https://t.co/CLMN3wNmeP

[Gary Marcus @GaryMarcus]

This so perfectly distills the tribal dynamic that stifles progress.

People choose sides—here with Hinton—without having any idea if the argument is valid. (As I discussed on my Substack, it wasn’t.)

[Carlos E. Perez @IntuitMachine]

Groq is a Radically Different kind of AI architecture

Among the new crop of AI chip startups, Groq stands out with a radically different approach centered around its compiler technology for optimizing a minimalist yet high-performance architecture. Groq's secret sauce is this compiler-first method that shuns complexity in favor of tailored efficiency.

At the heart of Groq’s architecture is an almost surprisingly bare-bones design that does away with unnecessary logic in favor of raw parallel throughput. The hardware itself is comparable to an ASIC – an application-specific integrated circuit finely tuned for machine learning. However, unlike a fixed-function ASIC, Groq leverages a custom compiler that can adapt and optimize across different models. It is this combination of a streamlined architecture and an intelligent compiler that sets Groq apart.

The key insight is that many AI chips stack components, like GPUs, that bring extraneous hardware and bloat. Groq returns to first principles, recognizing that machine learning workloads are about massive parallelism over simple data types and operations. By eliminating generic hardware and even concepts like locality, the design maximizes throughput and efficiency.

This is enabled by Groq’s compiler that sits between software frameworks like TensorFlow and the hardware. The compiler analyzes and optimizes neural network graphs, tailoring and mapping them to the underlying architecture for accelerated execution. It breaks computations into the smallest operations to unlock parallelism. The compiler also enables capabilities like batch size 1 inference that ensures all hardware is usefully leveraged.

Critically, Groq built its compiler before even finalizing the hardware design. The software insights directly informed the architecture. This co-design process allowed inference-specific optimization without legacy limitations. The compiler also provides deterministic guarantees of runtimes, enabling reliable scaling.

Together, the Groq compiler and architecture form a streamlined, robust engine for machine learning inference. The innovative compiler-first methodology allows custom optimization that balances flexibility with performance. Rather than chasing complexity, Groq realizes less can be more when software and hardware align – a compelling recipe as AI workloads continue evolving.

[Andrej Karpathy @karpathy]

"My benchmark for large language models"
https://t.co/YZBuwpL0tl

Nice post but even more than the 100 tests specifically, the Github code looks excellent - full-featured test evaluation framework, easy to extend with further tests and run against many LLMs.
https://t.co/KnmDD1AJci

E.g. for the 100 current tests on 7 models:
- GPT-4: 49% passed
- GPT-3.5: 30% passed
- Claude 2.1: 31% passed
- Claude Instant 1.2: 23% passed
- Mistral Medium: 25% passed
- Mistral Small 21% passed
- Gemini Pro: 21% passed

Also a huge fan of the idea of mining tests from actual use cases in the chat history. I think people would be surprised how odd and artificial many "standard" LLM eval benchmarks can be. Now... how can a community collaborate on more of these benchmarks... 🤔

[Shaun.AGI @agishaun]

⚠️RAG might be dead, after reading 58 pages of Genimi 1.5 Pro tech report. Here's my thoughts as AI founder,

1. Simple RAG system like similarity search with vector db will be dead. But more customized RAG will still live. The goal of RAG is mostly on retrieval relevant information. After reading the report, I am convinced LLM can do retrieval really really well.
2. RAG itself may not be dead totally, but 90% of people won't need it anymore. Most dataset can fit in 1M tokens. Just like OpenAI's assistant API, once Gemini API can handle large files, the only thing matters is the cost. However based on the report, 1.5 Pro's training cost and inference cost is much much lower than Gemini 1.0📉
3. LLM retrieval is multi-modal by default. It works on any data format, code, text, audio and video! Yes, gemini can retrieval key frame on video at 1 fps.🤯 When most RAG systems have hard time dealing with different data format, Gemini has solved it on the model level.
4. Retrieval != Reasoning. LLM is getting better at retrieval, but the reasoning is the holy grail.

If you want to read the tech report yourself -> https://t.co/21b1xuV7kF

[Carlos E. Perez @IntuitMachine]

1/n What in the world is Sora's "diffusion transformer model"?

A diffusion transformer model is a type of generative model for images, video, and other data that combines transformer architectures with diffusion probabilistic models. Here are some key details:

- Diffusion models work by taking real data and gradually adding noise to it over multiple steps, until it becomes an unstructured noise image. This process of adding noise is called the "forward diffusion" process.

- To generate new samples, diffusion models are trained to take a noisy image and predict how to remove some of the noise by estimating the difference between the image at the current noise level and the previous less noisy version. This is done repeatedly, removing more and more noise, until a realistic generated sample emerges.

- Transformers are neural network architectures that make use of self-attention to model long-range dependencies in sequential data. They have shown excellent performance in modeling text, images, video and other modalities.

- A diffusion transformer combines these two ideas. It uses a transformer encoder-decoder architecture to take in a noisy input image and predict the less noisy version at each step. The encoder encodes the noisy input, and the decoder generates the predictions of less noisy versions.

- Training involves minimizing the difference between the predicted less noisy images and the actual less noisy images from the forward diffusion process. At generation time, the model starts with random noise and repeatedly predicts less noise to generate a new sample.

- Diffusion transformers build in inductive biases like translation equivariance that make them very effective for image and video generation compared to other transformer variants. The patch-based processing integrates spatial information.

So in summary, a diffusion transformer leverages transformers' modeling power and diffusion modeling's noise schedule to generate high-quality, realistic image and video samples. The noise modeling allows precise control over the tradeoff between sample quality and diversity.

[Sebastian Raschka @rasbt]

While everyone is talking about Sora, there's a potential successor to LoRA (low-rank adaptation) called DoRA. Here's a closer look at the "DoRA: Weight-Decomposed Low-Rank Adaptation" paper: https://t.co/nDHYeoSUPf

LoRA is probably the most widely used parameter-efficient finetuning method for LLMs and vision transformers, and DoRA can be seen as an improvement or extension of LoRA that is built on top of it.

A brief LoRA recap: Assuming we have pretrained model weights W, LoRA uses low-rank matrices to approximate weight changes ΔW. I.e., in regular finetuning we have W' = W + ΔW, and in LoRA, we approximate ΔW with BA.

Now, the DoRA method first decomposes the pretrained weight matrix into a magnitude vector (m) and a directional matrix (V). Then, it takes the directional matrix V and applies standard LoRA to it, i.e., W' = m (V + ΔV)/norm = m (W + BA)/norm.

The motivation for developing this method is based on analyzing and comparing the LoRA and full finetuning learning patterns. They found that LoRA either increases or decreases magnitude and direction updates proportionally but seems to lack the capability of making only subtle directional changes as found in full finetuning. Hence, the researchers propose the decoupling of magnitude and directional components. In other words, their DoRA method aims to apply LoRA only to the directional component (while also allowing the magnitude component to be trained separably.)

Note that introducing the magnitude vector m in DoRA adds 0.01% more parameters than standard LoRA. However, across both LLM and vision transformer benchmarks, they found that DoRA even outperforms LoRA if the DoRA rank is halved, i.e., when DoRA only uses half the parameters of regular LoRA.

Overall, I am actually quite impressed by the results and need to toy with this method in practice. It should not be too big of a lift to upgrade a LoRA implementation to DoRA.

[Akari Asai @AkariAsai]

Introducing Self-RAG, a new easy-to-train, customizable, and powerful framework for making an LM learn to retrieve, generate, and critique its own outputs and retrieved passages, by using model-predicted reflection tokens.
📜: https://t.co/CbIdZeAmcR
🌐: https://t.co/j1kRrrS3Go