Daniel Koceja

[Karan Dalal @karansdalal]

LLM memory is considered one of the hardest problems in AI.

All we have today are endless hacks and workarounds. But the root solution has always been right in front of us.

Next-token prediction is already an effective compressor. We don’t need a radical new architecture. The missing piece is to continue training the model at test-time, using context as training data.

Our full release of End-to-End Test-Time Training (TTT-E2E) with @NVIDIAAI, @AsteraInstitute, and @StanfordAILab is now available.

Blog: https://t.co/woCpiIrq0T
Arxiv: https://t.co/3VkFlS3wx3

This has been over a year in the making with @arnuvtandon and an incredible team.

[Karan Dalal @karansdalal]

I want to highlight some of the coolest kernel work by @danielkoceja, specifically around making TTT layers fast for training and video generation.

TLDR: Our kernel does Tensor Parallel across Streaming Multiprocessors so we can efficiently train an RNN whose hidden state is a machine learning model.

Kernels like Flash Attention succeed by leveraging the GPU memory hierarchy. For some background:

- A GPU consists of multiple Streaming Multiprocessors (SMs) – similar to CPU cores.
- All SMs share a large but slower global memory called HBM.
- Each SM also has its own small but extremely fast on-chip memory, called SMEM.
- To achieve maximum efficiency, kernels load input data once into SMEM, compute there, and then write results directly back to HBM – avoiding frequent, costly transfers between these memory types.

However, TTT layers present a challenge for this approach. The hidden state of a TTT layer is itself a machine learning model, and it's too big to fit entirely into the small on-chip SMEM!

Interestingly, this resembles a common scenario we face when training large models. If a X-billion parameter model doesn't fit on one GPU, we typically use Tensor Parallelism to split the model across several GPUs. Why can't we do something similar for the TTT layer’s hidden state model?

We actually can – by applying the same sharding idea across the SMEMs of multiple SMs, treating each SM as the analogy of a GPU. NVIDIA’s H100 GPUs come with a unique "Distributed Shared Memory" feature, allowing SMs to do an "AllReduce" without touching the slower HBM. See our "update rule" in the attached figure.

This implementation trick provides a way to continue scaling the hidden state model of TTT layers, which we believe is the approach to generating even longer videos with more complex storylines.

As a general principle, if your model architecture can leverage standard Tensor Parallelism across GPUs, you can apply that same sharding strategy across SMs when using that model as the hidden state.