Jung Yi

Interested in Efficient World Models

[Jung Yi @jungyi3578]

@sethia_avik @reactorworld Thanks for the great post @sethia_avik — glad to see WorldKV used at @reactorworld!

[Avik Sethia @sethia_avik]

Wrote a blog on the great work done by @jungyi3578 and team and how we use it at @reactorworld

[Jung Yi @jungyi3578]

Great to see WorldKV running at @reactorworld. KV cache management will be one of the most important problems to solve in world models — this article is worth a read.

[Avik Sethia @sethia_avik]

Wrote a blog on the great work done by @jungyi3578 and team and how we use it at @reactorworld

[Avi Chawla @_avichawla]

A tricky LLM interview question:

You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces.

So you add KV cache compression and evict 90% of the cached tokens.

VRAM usage stays as is and GPU still runs out of memory.

Why?

(answer below)

Evicting 90% of the KV cache can free almost none of the memory it was using.

This sounds counterintuitive, but it follows directly from how production servers store the cache today.

The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues.

This is the dominant memory cost for reasoning models.

If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU.

One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it.

But this does not solve the memory problem yet.

The reason is paged attention, which is the memory manager behind vLLM and most production servers.

Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens.

This block returns to the allocator only when every slot inside it is empty.

Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks...

...so despite eviction, almost every block is left with at least some survivor tokens.

For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token.

This means the allocator frees almost nothing.

Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout.

Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order.

Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds.

This introduces another bookkeeping cost that an in-order layout inherently avoids.

So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server.

There's another problem.

Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected).

But fast attention kernels used in production, like FlashAttention, never save those scores.

They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast.

So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide.

NVIDIA published a method called TriAttention to solve both these problems.

It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters.

For the memory problem, it runs a compaction pass every 128 decoded tokens.

The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order.

On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory.

KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens.

You can find the NVIDIA write-up here: https://t.co/ZwXv7VezVu

I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching.

Read it below.

[Jung Yi @jungyi3578]

Umm… I didn’t realize I had to change my DM settings to receive messages 😅

If you’re interested in chatting about Efficient Video World Models at #ICML, feel free to DM me — they’re really open now! ☕️

[Jung Yi @jungyi3578]

I'll be presenting Deep Forcing at #ICML in Seoul!
My DMs are open — I'd especially love to grab a coffee and chat about efficient video world models ☕

https://t.co/s5PTbw3vKd

[Turing Post @TheTuringPost]

Must-read research of the week

▪️ ScientistOne
▪️ SkillOpt
▪️ MUSE-Autoskill
▪️ Do Language Models Need Sleep?
▪️ OmniRetrieval
▪️ Vector Policy Optimization
▪️ Gamma-World
▪️ OpenComputer
▪️ Personalize-then-Store
▪️ WorldKV
▪️ AutoResearchClaw
▪️ Qwen-VLA
▪️ CUA-Gym

Find the full list and the most important AI news of the week here: https://t.co/UC1IeiRL8y

[Jung Yi @jungyi3578]

I'll be presenting Deep Forcing at #ICML in Seoul!
My DMs are open — I'd especially love to grab a coffee and chat about efficient video world models ☕

https://t.co/s5PTbw3vKd

[Jung Yi @jungyi3578]

🚨The code for our paper “Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression” is now available! #KAIST

🎥 A training-free method for minute-long video generation in autoregressive video diffusion, mitigating error accumulation and motion degradation during long rollouts.

✅ Deep Sink — maintains a substantially enlarged attention sink (~50% of the KV cache) with temporal RoPE adjustment.

✅ Participative Compression — performs importance-aware KV cache pruning based on which tokens are actually attended, evicting redundant and degraded tokens.

✅ Plug-and-play KV-cache management — improves long-video stability without additional training; integrates into existing AR diffusion pipelines (e.g., Self Forcing, Causal Forcing).

✅ Broader applicability — also works for Interactive Prompting, World Models (Check out our results on the project page!)

Project Page: https://t.co/o7lpY7p4q1

Paper: https://t.co/nJKbt8Xk8O

#AutoregressiveVideoDiffusion #WorldModel #LongVideoGen #SelfForcing #DeepForcing

[Jung Yi @jungyi3578]

🚨The code for our paper “Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression” is now available! #KAIST

🎥 A training-free method for minute-long video generation in autoregressive video diffusion, mitigating error accumulation and motion degradation during long rollouts.

✅ Deep Sink — maintains a substantially enlarged attention sink (~50% of the KV cache) with temporal RoPE adjustment.

✅ Participative Compression — performs importance-aware KV cache pruning based on which tokens are actually attended, evicting redundant and degraded tokens.

✅ Plug-and-play KV-cache management — improves long-video stability without additional training; integrates into existing AR diffusion pipelines (e.g., Self Forcing, Causal Forcing).

✅ Broader applicability — also works for Interactive Prompting, World Models (Check out our results on the project page!)

Project Page: https://t.co/o7lpY7p4q1

Paper: https://t.co/nJKbt8Xk8O

#AutoregressiveVideoDiffusion #WorldModel #LongVideoGen #SelfForcing #DeepForcing

[Jung Yi @jungyi3578]

The code for WorldKV is now public!
https://t.co/0WsFvhfCkV

[Jung Yi @jungyi3578]

🚨Our paper “WorldKV: Efficient World Memory with World Retrieval and Compression” is now available!

🎥 A training-free framework for efficient world memory in autoregressive video world models.

Large World Models (LWMs) need efficient KV cache management at inference — full KV cache attention cost and KV cache footprint aren't affordable, even on a B200. Sliding-window inference is fast but forgets; full-history attention remembers but isn't real-time. WorldKV breaks this trade-off with two training-free components:

✅ World Retrieval — retrieves only viewpoint-relevant KV caches via camera/action correspondence, instead of attending to the full history.

✅ World Compression — key-key similarity-based pruning evicts redundant/overlapping tokens while preserving newly revealed regions and dynamic object regions.

✅ Plug-and-play — integrates into existing AR video world model pipelines without any fine-tuning (LingBot-World-Fast, Matrix-Game-2.0, Inspatio-World).

Results — ½ VRAM footprint and ~2× throughput on long rollouts; matches Full KV memory fidelity on LingBot-World-Fast (14B), and even surpasses it on Matrix-Game-2.0 (1.3B). We also show that full-history attention isn't always optimal when much of the KV cache is irrelevant or redundant.

Project Page: https://t.co/cyiVa0LXAp

Paper: arXiv:2605.22718

#AutoregressiveVideoDiffusion #WorldModel

[Jung Yi @jungyi3578]

🚨Our paper “WorldKV: Efficient World Memory with World Retrieval and Compression” is now available!

🎥 A training-free framework for efficient world memory in autoregressive video world models.

Large World Models (LWMs) need efficient KV cache management at inference — full KV cache attention cost and KV cache footprint aren't affordable, even on a B200. Sliding-window inference is fast but forgets; full-history attention remembers but isn't real-time. WorldKV breaks this trade-off with two training-free components:

✅ World Retrieval — retrieves only viewpoint-relevant KV caches via camera/action correspondence, instead of attending to the full history.

✅ World Compression — key-key similarity-based pruning evicts redundant/overlapping tokens while preserving newly revealed regions and dynamic object regions.

✅ Plug-and-play — integrates into existing AR video world model pipelines without any fine-tuning (LingBot-World-Fast, Matrix-Game-2.0, Inspatio-World).

Results — ½ VRAM footprint and ~2× throughput on long rollouts; matches Full KV memory fidelity on LingBot-World-Fast (14B), and even surpasses it on Matrix-Game-2.0 (1.3B). We also show that full-history attention isn't always optimal when much of the KV cache is irrelevant or redundant.

Project Page: https://t.co/cyiVa0LXAp

Paper: arXiv:2605.22718

#AutoregressiveVideoDiffusion #WorldModel

[Jung Yi @jungyi3578]

🚨The code for our paper “Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression” is now available! #KAIST

🎥 A training-free method for minute-long video generation in autoregressive video diffusion, mitigating error accumulation and motion degradation during long rollouts.

✅ Deep Sink — maintains a substantially enlarged attention sink (~50% of the KV cache) with temporal RoPE adjustment.

✅ Participative Compression — performs importance-aware KV cache pruning based on which tokens are actually attended, evicting redundant and degraded tokens.

✅ Plug-and-play KV-cache management — improves long-video stability without additional training; integrates into existing AR diffusion pipelines (e.g., Self Forcing, Causal Forcing).

✅ Broader applicability — also works for Interactive Prompting, World Models (Check out our results on the project page!)

Project Page: https://t.co/o7lpY7p4q1

Paper: https://t.co/nJKbt8Xk8O

#AutoregressiveVideoDiffusion #WorldModel #LongVideoGen #SelfForcing #DeepForcing