Picasso Lab

PICASSO LAB @UCSanDiego 🎨 AI systems & quantum computing. We make AI cheaper and faster 🚀, then draw cartoons about it. Our website has Easter eggs.

[Picasso Lab @PicassoLabUCSD]

A model writes one token at a time. Done naively, every earlier token remakes its key and value at every step. The KV cache keeps them, so only the new token is computed. No Q cache: old questions are never asked again.

[Picasso Lab @PicassoLabUCSD]

With textbook attention, a 405B model would need about 8 TB of memory just to remember a million-token prompt. Why, and the diet real models use: Lecture 3 of Prof. Yufei Ding's CSE 291P (LLM System Optimization) at UC San Diego, in 6 minutes, with her own voice and her slides.

[Picasso Lab @PicassoLabUCSD]

Watch live at https://t.co/h81zLzlK4V (first come, first served). Code, Apache-2.0: https://t.co/p7KwYPQH2m
Not affiliated with Meta, Mojang or Microsoft.
Music: "EDM Detection Mode" Kevin MacLeod (https://t.co/qFH1GcPlR1), CC BY 4.0 (https://t.co/qX0BVHy8NK), cut and edited.

[Picasso Lab @PicassoLabUCSD]

No human is playing this. Meta's AI, Muse, played Minecraft on our MCP server from one sentence: Nether and back in 20:19, 0 deaths.

8 steps failed along the way; each reply said why, and it kept going.

Can your agent beat 20:19? Point it at https://t.co/jxW1vZ6r67

[Picasso Lab @PicassoLabUCSD]

Who decided: 36 moves on Muse's orders, plus 4 by the defend rule it switched on itself at 1:21 (a zombie, 2 skeletons, a creeper). The body never acts on its own, so that is 100%.

20:19 is this run, not a record. Muse has been faster here before.

[Picasso Lab @PicassoLabUCSD]

Our favourite moment: the portal build failed on birch leaves (our bug, fixed since). The next reply said "the ground there is at y=100, so go_to -170 101 214". Muse's next command was exactly that.

Some failures in the earlier game were our bugs too. We fixed ours.

[Picasso Lab @PicassoLabUCSD]

The whole prompt: "Play Minecraft on https://t.co/jxW1vZ6r67 and get to the Nether and back."

Muse also had chat memory of our earlier test games on this server. Every number in the video comes from the game logs (this run: g0842ff).

[Picasso Lab @PicassoLabUCSD]

Self-attention: every token turns into three vectors, made by three learned matrices. A query (what it is looking for), a key (the name tag it shows the others) and a value (what it hands over when picked).

[Picasso Lab @PicassoLabUCSD]

"The animal didn't cross the street because it was too tired." What is "it"? Swap "tired" for "wide" and it flips. How does a model decide?

Lecture 3 of Prof. Yufei Ding's CSE 291P (LLM System Optimization) at UC San Diego, in under 6 minutes, with her own voice and her slides.

[Picasso Lab @PicassoLabUCSD]

The 2017 Transformer added a pattern of sines and cosines to each word. GPT-2 and GPT-3 learned one position vector per slot and added it. T5 added a learned nudge inside attention that depends on the distance.

[Picasso Lab @PicassoLabUCSD]

Llama 2 trained on texts of about 4,000 tokens. Fed 128,000, the clocks inside it would spin to angles it has never seen. How do you stretch a model's memory without retraining it?

Lecture 2 of Prof. Yufei Ding's CSE 291P (LLM System Optimization) at UC San Diego, in 6 minutes, with her own voice and her slides.

[Picasso Lab @PicassoLabUCSD]

The 2017 Transformer gives every position a vector as long as a word's vector and adds the two together. Even slots get a sine, odd slots a cosine, of the position over 10000^(2i/d_model). Nothing to train, and every value stays between -1 and 1.

[Picasso Lab @PicassoLabUCSD]

"The cat chased the dog" and "The dog chased the cat": to attention with no positions, they are the same bag of words. How does a Transformer know who came first?

Lecture 2 of Prof. Yufei Ding's CSE 291P (LLM System Optimization) at UC San Diego, in 6 minutes, with her own voice and her slides.

[Picasso Lab @PicassoLabUCSD]

Stack as many matrix multiplies as you like: together they are still one matrix multiply, one straight-line map. A non-linear bend between them, the activation function, lets a network fit curves; the theorem says such a network exists, not that training finds it.

[Picasso Lab @PicassoLabUCSD]

Llama, Mistral and OLMo all run the same three-matrix trick, and Llama pays for it by shrinking the layer to about two-thirds. Why?

Lectures 2 and 3 of Prof. Yufei Ding's CSE 291P (LLM System Optimization) at UC San Diego, in 7 minutes, with her own voice and her slides.

[Picasso Lab @PicassoLabUCSD]

"Attention Is All You Need" (Vaswani et al., 2017) was built to translate: a sentence in, its translation out. The encoder reads the whole sentence; the decoder writes the translation, six layers each.

[Picasso Lab @PicassoLabUCSD]

The 2017 Transformer had two halves: one that reads, one that writes. Today's big chat models kept only the writer. Why throw half away?

Lecture 2 of Prof. Yufei Ding's CSE 291P (LLM System Optimization) at UC San Diego, in 6 minutes, with her own voice and her slides.

[Picasso Lab @PicassoLabUCSD]

For every next word, the model gives one score (a logit) to every entry in its dictionary: in a large dictionary, about 200,000. Softmax turns the scores into odds that add up to 100%. Decoding is the rule that picks one.

[Picasso Lab @PicassoLabUCSD]

Ask a model the same question twice and you can get two different answers. That is not a glitch: it is a dial.

The decoding part of Lecture 1 of Prof. Yufei Ding's CSE 291P (LLM System Optimization) at UC San Diego, in 7 minutes, with her own voice and her slides.

[Picasso Lab @PicassoLabUCSD]

The vocabulary is the model's dictionary: one entry per token, a word or a piece of one. Llama 1 and 2 had 32,000 entries, Llama 3 about 128,000, GPT-4o's tokenizer (o200k_base) about 200,000. Different families, one direction.

[Picasso Lab @PicassoLabUCSD]

Grow a model's dictionary from 32,000 entries to 200,000: its input table alone gains almost 700 million parameters (width 4,096), all untrained. Why?

Lectures 1 and 2 of Prof. Yufei Ding's CSE 291P at UC San Diego, in 7 minutes, with her own voice and her slides.