Kaustab Pal | LLM Systems

Building task-specialized LLM systems from first principles. Experiments, deep dives & build log.

[Kaustab Pal | LLM Systems @kaustabpal]

@ErickSky @Sentdex waiting for your GLM video in this one.

[Kaustab Pal | LLM Systems @kaustabpal]

Inspired by @DubThinkDev’s Marathon spinner, I vibe-coded this matrix organism in python. It represents the thinking loop of a LLM and runs entirely in the terminal.
Will try to replace Pi’s thinking text with this next.

https://t.co/lrG49pLu4S

[Kaustab Pal | LLM Systems @kaustabpal]

POV: Your local LLM is thinking and it looks like this gorgeous hand-coded shader butterfly.
Someone make this real please.

[Kaustab Pal | LLM Systems @kaustabpal]

POV: Your local LLM is thinking and it looks like this gorgeous hand-coded shader butterfly.
Someone make this real please.

[Abby Welsh @DubThinkDev]

Fun fact: the Marathon spinner is actually a hand-coded shader! It has to display before the engine loads any textures or videos, so it's entirely procedurally animated and rendered by a shader I wrote for it using SDFs, procedural noise, and a pile of math for the animation :)

[Kaustab Pal | LLM Systems @kaustabpal]

My DGX Spark arrived today.

Instead of studying vLLM, I spent the evening setting it up. Following @MiaAI_lab’s recipe, I was able to run Qwen3.6 35B smoothly using the Pi harness, and it was blazing fast.

To be honest, there was no noticeable difference in decode speed between local Qwen and Codex 5.5, and the quality of the answers was pretty good too. I used Qwen to set up Syncthing on my Spark to sync files with my Mac, and it just worked without any errors.

I’ll spend the next couple of days tinkering with the setup, testing different models and harnesses, and of course doing some wire management on my desk. It’s safe to say I now get the hype behind the Spark. ✨

[Kaustab Pal | LLM Systems @kaustabpal]

Day 2: PagedAttention

For each token, there is a query (qi​), key (kj​), and value (vj​) vector. In standard attention, to find the output for token i, its query vector is multiplied by all previous key vectors to compute the attention scores. The final output is the weighted average of the corresponding value vectors.

In PagedAttention, the process is transformed into a blockwise computation. The KV cache of each sequence is partitioned into blocks, each block storing key and value vectors for a fixed number of tokens. These blocks are stored non-contiguously across the physical memory space.

During computation, the query vector qi​ is multiplied with a specific block of key vectors to produce a row vector of attention scores for that block.
These scores are then multiplied by the corresponding block of value vectors to obtain a partial output.
The final output is the sum of these partial results across all blocks. This mechanism allows the KV cache to be managed flexibly in non-contiguous physical memory.

During computation, the PagedAttention kernel identifies and fetches each block separately using the Block Table. This design is what enables vLLM’s flexible, paged memory management.

[Kaustab Pal | LLM Systems @kaustabpal]

Day 1: The problem vLLM is solving

LLMs generate tokens one at a time autoregressively based on the input prompt and the sequence of the output so far. As a result, Each step requires loading the Key-Value (KV) cache for all previous tokens from GPU memory (VRAM) to its processors. Loading data from GPU's VRAM to the processor is a time consuming operation and the processor has to wait for the data to be loaded from the VRAM. This reduces the serving throughput. Which is why memory space for each request should be managed efficiently.

Existing systems store the KV cache in contiguous memory but since the cache grows dynamically, they must pre-allocate a chunk based on a request's maximum possible length.

This leads to two problems:

1. Preallocating a chunk of memory for the KV cache will often lead to internal fragmentations as the entire preallocated chunk of memory might never be used.

2. For different requests, the preallocation chunk might be different for each requests. This leads to external fragmentation. i.e. small gaps between reserved memory chunks that are too small to be allocated to any new request.

Additionally, existing systems cannot share memory between different requests. Often times two different requests same sequences and as a result will share part of the existing KV cache. But since each request gets their own allocated memory chunk they cannot reuse the KV cache of shared sequences between requests and the KV-cache needs to be computed from scratch.

vLLM solves the above problems using PagedAttention which is inspired from OS virtual memory. PagedAttention partitions the KV cache into fixed-size blocks that can be stored in non-contiguous physical memory, using a Block Table to map them. Each block contains the attention key and values for a fixed number of tokens. This eliminates external fragmentation because all blocks are the same size, and it reduces internal fragmentation to near-zero by allocating memory only as needed.

vLLM also handles KV cache reuse by allowing multiple sequences to point to the same physical blocks in memory through its Block Table.

[Kaustab Pal | LLM Systems @kaustabpal]

When computers first started becoming mainstream, they were huge, expensive machines that mostly hobbyists and programmers would buy and tinker with. Over time, computers kept getting smaller, cheaper, and more capable, and now almost everyone carries one in their pocket.

I think we are at a similar early stage with LLMs. The hardware is still expensive, the most capable models are still the larger ones, and for now, mostly hobbyists and developers are the ones trying to own the hardware and run LLMs locally.

But just as computers evolved, the LLM ecosystem will evolve too. Smaller models will become much, much faster and more capable, and eventually people will be walking around with Walkman-sized devices running their own personal LLMs. A cassette player for your mind.

We’re just getting started.

[Kaustab Pal | LLM Systems @kaustabpal]

When computers first started becoming mainstream, they were huge, expensive machines that mostly hobbyists and programmers would buy and tinker with. Over time, computers kept getting smaller, cheaper, and more capable, and now almost everyone carries one in their pocket.

I think we are at a similar early stage with LLMs. The hardware is still expensive, the most capable models are still the larger ones, and for now, mostly hobbyists and developers are the ones trying to own the hardware and run LLMs locally.

But just as computers evolved, the LLM ecosystem will evolve too. Smaller models will become much, much faster and more capable, and eventually people will be walking around with Walkman-sized devices running their own personal LLMs. A cassette player for your mind.

We’re just getting started.

[Kaustab Pal | LLM Systems @kaustabpal]

@MiaAI_lab I think the Gigabyte one has the best cooling. I have personally ordered the MSI variant because I liked the aesthetic of that one. Waiting for delivery. But aren’t you guys worried of the devices becoming obsolete if NVIDIA stops supporting DGX OS after 2 years?

[Gill @gurtej__gill_]

This new paper from Stanford, Berkeley and NVIDIA feels like a massive piece of the puzzle when it comes to test time compute.

They’re proposing LLM as a Verifier, arguing that verification is the next major scaling axis we need to look at.

Instead of treating LLM judges like a black box that spits out a rigid discrete score, they extract the token logits to calculate a continuous probabilistic expectation.

It sounds nuanced, but it completely fixes the "tie rate" problem when you're trying to compare two complex long horizon agent trajectories.

By scaling up the score granularity, running repeated evaluations and decomposing the criteria, they get incredibly calibrated results.

The performance gains are wild: SOTA across the board, including hitting 86.5% on Terminal-Bench 2.0 (beating GPT-5.5) and crushing SWE-Bench Verified at 78.2%.

And this isn't just theoretical.

Because the feedback is so fine grained, they’re using it as a dense reward signal to massively speed up RL efficiency (like GRPO and SAC) and they’ve already built a practical extension for Claude Code to track agent progress in real time.

Definitely a work worth reading by Jacky Kwok, Chelsea Finn, Ion Stoica and the rest of the team.

Read the full paper here:
https://t.co/t8Pps8qynx

[Kaustab Pal | LLM Systems @kaustabpal]

Day 1: The problem vLLM is solving

LLMs generate tokens one at a time autoregressively based on the input prompt and the sequence of the output so far. As a result, Each step requires loading the Key-Value (KV) cache for all previous tokens from GPU memory (VRAM) to its processors. Loading data from GPU's VRAM to the processor is a time consuming operation and the processor has to wait for the data to be loaded from the VRAM. This reduces the serving throughput. Which is why memory space for each request should be managed efficiently.

Existing systems store the KV cache in contiguous memory but since the cache grows dynamically, they must pre-allocate a chunk based on a request's maximum possible length.

This leads to two problems:

1. Preallocating a chunk of memory for the KV cache will often lead to internal fragmentations as the entire preallocated chunk of memory might never be used.

2. For different requests, the preallocation chunk might be different for each requests. This leads to external fragmentation. i.e. small gaps between reserved memory chunks that are too small to be allocated to any new request.

Additionally, existing systems cannot share memory between different requests. Often times two different requests same sequences and as a result will share part of the existing KV cache. But since each request gets their own allocated memory chunk they cannot reuse the KV cache of shared sequences between requests and the KV-cache needs to be computed from scratch.

vLLM solves the above problems using PagedAttention which is inspired from OS virtual memory. PagedAttention partitions the KV cache into fixed-size blocks that can be stored in non-contiguous physical memory, using a Block Table to map them. Each block contains the attention key and values for a fixed number of tokens. This eliminates external fragmentation because all blocks are the same size, and it reduces internal fragmentation to near-zero by allocating memory only as needed.

vLLM also handles KV cache reuse by allowing multiple sequences to point to the same physical blocks in memory through its Block Table.

[Kaustab Pal | LLM Systems @kaustabpal]

Next, I’ll be doing a deep dive into the vLLM paper to understand how these ideas are implemented in practice.

https://t.co/WPBKdetq6a

[Kaustab Pal | LLM Systems @kaustabpal]

Missed the chance to call it Doom Slayer.

[Liquid AI @liquidai]

Today we release Antidoom, an open-source method that removes a common failure mode in reasoning models: the doom loop.

Doom-loop rates before and after, with eval scores up across the board:

> Early LFM2.5-2.6B checkpoint: 10.2% → 1.4%

> Qwen3.5-4B: 22.9% → 1% (greedy sampling)

🧵

[Kaustab Pal | LLM Systems @kaustabpal]

@HarveenChadha @ankitdotme Why not go local? Investing in a good system that can serve GLM5.2 to all the internal engineers. Won’t that reduce the cost in the long run?

[Kaustab Pal | LLM Systems @kaustabpal]

A love letter from @GoogleDeepMind

[Omar Sanseviero 🇫🇷 @RAISE Paris @osanseviero]

Happy to share we just published Gemma 4 technical report! Take a look

[Lilian Weng @lilianweng]

new post on harness engineering for AI self-improvement: https://t.co/ZYvGfVs61k

It is hard to forecast how much the future of RSI will rely on harnesses. Likely harness engineering will evolve in the direction of self-improvement and enable auto-research, and, in turn, smarter models keeps harnesses simple.

Even when many harness improvement get eventually internalized into core model, the need to specify goals and context will not disappear.

[Kaustab Pal | LLM Systems @kaustabpal]

Next, I’ll be doing a deep dive into the vLLM paper to understand how these ideas are implemented in practice.

https://t.co/WPBKdetq6a

[Kaustab Pal | LLM Systems @kaustabpal]

Day 0: As a start I wanted to properly understand why we need to use an inference engine and not just call the models through HF transformers.

The first reason is efficient batching. LLM inference is different from normal neural network inference because output generations have variable token lengths.

In a naive static batch, if one request finishes early, the GPU may continue processing only the remaining requests while the freed slot stays unused.

Ideally, as soon as one request finishes, another request should enter the active batch.

This is called continuous batching. Inference engines like vLLM maintain an active batch and dynamically schedule requests so GPU compute is used more efficiently.

The second reason is KV cache management. In a agentic workflow where the LLM works in a loop with large repeated prefixes and relatively short outputs, only a small tail of the context changes each step. Which means for the repetitive part we need to calculate the KV pair once and reuse it.

While transformers gives us the low-level KV-cache mechanism, but it does not provide a full cache-management system. It does not automatically handle prefix reuse across requests, cache eviction, cache sharing between similar prompts, or memory-aware scheduling when many requests are running at the same time.

Inference engines like vLLM add this missing runtime layer. They manage the KV cache as a shared GPU memory resource, which makes repeated agent loops and multi-request serving more efficient.

Apart from batching and KV-cache management, vLLM also includes several other optimisations such as optimized kernels, execution graphs, support for multiple quantization formats, streaming APIs, and distributed inference. I am yet to go deeper into these parts, but the key point is that vLLM is not just running the model; it is managing the full inference workload.

In short, if we just want to use an LLM to generate an output for a single input, almost like calling a Python function, then Transformers is usually enough.

But if we want to build a complete serving stack that can handle multiple requests from multiple users, keep the GPU efficiently utilised, manage KV-cache memory, support continuous batching, and optimise throughput, then an inference engine like vLLM is the better choice.

[Kaustab Pal | LLM Systems @kaustabpal]

I will be doing a deep dive on vLLM this week. Will keep updating my progress here.

[Kaustab Pal | LLM Systems @kaustabpal]

I will be doing a deep dive on vLLM this week. Will keep updating my progress here.

[Aritra 🤗 @ariG23498]

vLLM deep dive anyone? 🧑‍🍳

[Kaustab Pal | LLM Systems @kaustabpal]

@ariG23498 I am in. Will be spending this week trying to learn as much as I can about vLLM. Will be updating my progress in my account.

[Xudong Han @Xudong07452910]

CMU 这门 NLP 课最硬核的地方：第一份作业就让你从零手搓一个 LLaMA。

卡内基梅隆大学公开了 2026 春季《Advanced NLP》课程，教授是 Sean Welleck。整套资源包括完整课程主页、课件、视频和配套代码，内容从 Tokenizer、Transformer、语言模型基础，一路讲到 RAG、多模态、RLHF、MoE、长上下文和 test-time scaling。

这门课最有价值的地方，不仅是教你怎么调用大模型 API，而且把大模型能力背后的工程路径也拆开给你看。

从「Build Your Own LLaMA」这种手搓模型作业，到后面的推理、效率、评测和 Agent，它训练的其实是一种更底层的能力：你能不能理解模型为什么能工作，为什么会变强，为什么有时会失败。

这也很符合现在 AI 的技术拐点。

过去大家更关心参数规模，谁的模型更大；现在越来越多关键进展，开始发生在推理侧、效率侧和系统侧：怎么让模型更会思考，怎么调度专家，怎么检索外部知识，怎么在有限算力下获得更好的输出。

所以这门课不只是 NLP 进阶课，更像一份 2026 年大模型研究者的训练路线图。

AI 时代真正的分水岭，可能不只是会不会使用模型。

更关键的是：

你停在应用层，还是愿意继续往下走到架构层。

Home: https://t.co/U96QszHIja
Youtubu: https://t.co/05dlefgqbe
Code: https://t.co/nqAsjPwatQ

[Kaustab Pal | LLM Systems @kaustabpal]

@prateek_0041 Wait a min. Did you figure out a way to switch your Mac keys?