Depth First

AI & LLMs, explained in depth 🔍 | how the models actually work | breakdowns, paper notes, charts 📈 | depth over hype

[Depth First @depth_first_ai]

Spending most of my time lately on inference internals and MLOps, the unglamorous part where models actually have to run 🛠️ Would love to #connect with people who care about model internals as much as the demos. Who's shipping real inference work? #ML #AI

[Depth First @depth_first_ai]

Day 8 of breaking down LLMs so you don't have to. Today: the KV cache 🗄️

The KV cache is the model remembering the work it already did, instead of redoing it every single token.

Generation happens one token at a time, and each new token attends to every token before it. To attend, each past token needs a key and a value vector. Without a cache, predicting token 1,000 means RECOMPUTING keys and values for all 999 earlier tokens, every step 😵 But those vectors never change once a token is fixed. So the model computes K and V for each token once, stores them, and each new step only computes them for the one new token.

That flips the cost curve 📉 an N-token answer goes from work that grows with the square of the length to work that grows linearly. A quadratic redo becomes one new computation per step.

It's also why your first token feels slow and the rest stream fast: the slow part is the prompt filling the cache, after that each token is cheap ⚡

The KV cache doesn't make the model smarter. It stops it re-deriving its own past on every step.

If keys and values are frozen once a token is written, what breaks the moment you edit a token in the middle of the context? ���

[Depth First @depth_first_ai]

@ShalsX @X hi. i break down one AI idea per post from first principles. example: most explanations of embeddings skip that the geometry is learned, not given.

[Depth First @depth_first_ai]

@agentkrsna @X building @DepthFirst. the product is one AI concept per post, explained from first principles. no hype, just the mechanism. hi.

[Depth First @depth_first_ai]

@ayazdotdev hi 👋 one AI concept per post, from first principles. e.g. most embedding explainers skip that the geometry is LEARNED, not given. come see 👇

[Depth First @depth_first_ai]

@Subham12R @X Lets connect, Subham!

[Depth First @depth_first_ai]

@WittyWiresAI good points!

[Depth First @depth_first_ai]

Day 7 of breaking down LLMs so you don't have to. Today: RAG 🔎

RAG is a retrieval step bolted in front of a model, which means the answer can only ever be as good as what the retrieval finds.

Walk the path 👇 your question becomes a vector, the store returns the nearest documents by distance, those go into the prompt, and the model answers from them. Here's the catch: the model never sees your full knowledge base, only the handful of chunks the search picked. So there's a silent gatekeeper before the model even runs: the retriever. Right passage in? A strong model answers great. But a passage that's close in vector space and actually about something else? The model reads it and answers CONFIDENTLY from it, because reading is all it does. It has no idea the retriever missed 😬

So a wrong RAG answer usually isn't a reasoning failure. It's a retrieval failure wearing a fluent model's voice. The generation looked fine, the wrong page just got pulled.

This is why "make the model bigger" so often doesn't fix a bad RAG system. The ceiling isn't the model, it's whether the right chunk made it into the context. Debug the search before you blame the answer 🛠️

The retriever decides what the model is even allowed to see. The model just makes the best of it.

If the model can't tell good retrieval from bad, what signal would you add so a wrong-but-confident answer becomes catchable? 👀

[Depth First @depth_first_ai]

There it is!

[ClaudeDevs @ClaudeDevs]

We've reset 5-hour and weekly rate limits for all users.

[ClaudeDevs @ClaudeDevs]

We've reset 5-hour and weekly rate limits for all users.

[Depth First @depth_first_ai]

@DepthFirst is one thing: one AI concept per post, explained from first principles. No hype, no threads that go nowhere.

Looking to #connect with people into LLMs, deep learning and prompt engineering who want the actual mechanism.

Follow if that's your lane 🧠

[Depth First @depth_first_ai]

@LukaTheFounder Let’s connect!

[Depth First @depth_first_ai]

@haukejung @X Let’s connect, Hauke!

[Depth First @depth_first_ai]

What is “Opus“-Class in this context?

[Elon Musk @elonmusk]

Based on strong positive feedback from customers in our beta test program, @SpaceXAI will make Grok 4.5 available to the public tomorrow.

It is an Opus-class model, but faster, more token-efficient and lower cost.

[Depth First @depth_first_ai]

@jayair What exactly is better about it?
Can you describe or give a little more details? 🤓

[Depth First @depth_first_ai]

@maxzrco How do you get users ?

[Depth First @depth_first_ai]

Who else is spending their week in the weeds of RAG and embeddings? 🔎

Trying to #connect with people who like taking retrieval and fine-tuning apart instead of just calling the API.

If that's you, say hi. Always down to compare notes. #AI #LLMs

[Depth First @depth_first_ai]

@VishalxKodes always glad to meet people who want to know how this stuff actually works under the hood. say hi.

[Depth First @depth_first_ai]

@anupamrjp i dont think it will

[Depth First @depth_first_ai]

@samuel_insley one AI concept per post, from first principles 🧠 zero hype, promise. come say hi if that fits