Arjun Vikram

proud arch linux user @openai

[Arjun Vikram @arjvik]

Proud to share what I've been working on for the past year, "Learning to (Learn at Test Time)"!

Our new architecture trains a model to "learn" from its context, replacing Attention's costly KV cache with an expressive hidden state: the weights of a ML model!🤯

🧵by @karansdalal

[Karan Dalal @karansdalal]

I’m excited to share a project I’ve been working on for over a year, which I believe will fundamentally change our approach to language models.

We’ve designed a new architecture, which replaces the hidden state of an RNN with a machine learning model. This model compresses context through actual gradient descent on input tokens. We call our method “Test-Time-Training layers.”

TTT layers directly replace attention, and unlock linear complexity architectures with expressive memory, allowing us to train LLMs with millions (someday billions) of tokens in context.

Our instantiations, TTT-Linear and TTT-MLP, both match or beat the strongest Transformers and Mamba. Arxiv: https://t.co/3eEenKB17s

[Noam Shazeer @NoamShazeer]

I’m excited to share that I’ll be joining OpenAI and look forward to working with the exceptional team there.

It was a difficult decision to move on. I’m incredibly proud of the amazing team at Google and everything we’ve built together. It has been an honor and a pleasure to work with all of you.

[Arjun Vikram @arjvik]

@bernhardsson @_gongy and it’s a stick shift!

[Fulcrum @fulcrum_inc]

🚨 We're open-sourcing Druids, a library for coordinating and deploying coding agents across machines.

Our beta users have used Druids to work on open math problems, conduct ML "autoresearch," and make software faster.

[devansh @devanshpandey]

there is much work to do. in the next few months we need to train 100x larger models on 200x as much data as we have done before, scale RL to have millions of agents learning things from the internet, and use the computer use prior to build the first models that can scalably learn. there's no team i'd rather be doing this with. if you're interested in being a part of SI's next chapter, i'd be very excited to chat.

[Standard Intelligence @si_pbc]

Computer use models shouldn't learn from screenshots.

We built a new foundation model that learns from video like humans do. FDM-1 can construct a gear in Blender, find software bugs, and even drive a real car through San Francisco using arrow keys.

[Yejin Choi @YejinChoinka]

Incredibly excited to share our latest work with @NVIDIAAI that has been over a year in the making: 🔥End-to-End Test-Time Training 🔥

Automated researchers need to compress years of experience into intuition 🧠– far too much to just dump into context. But how do humans do research? We are continual learners 👩‍🎓📚

Let’s make AI do the same 🤖📚: as the model reads through long context at test time, keep training to update the weights!

[Arnuv Tandon @arnuvtandon]

Excited to release a new paper today: “End-to-End Test-Time Training for Long Context”.

Our method, TTT-E2E, enables models to continue learning at test-time via next-token prediction on the given context – compressing context into model weights.

For our main result, we extend 3B parameter models from 8K to 128K. TTT-E2E scales with context length like full attention without maintaining keys and values for every token in the sequence.
With linear-complexity, TTT-E2E is 2.7x faster than full attention at 128K tokens while achieving better performance.

Paper: https://t.co/qOi2k3aIKd
Code: https://t.co/nyko0Ap5H6

[Arjun Vikram @arjvik]

Test-Time Training lives on!

[Karan Dalal @karansdalal]

Our new paper, “End-to-End Test-Time Training for Long Context,” is a step towards continual learning in language models.

We introduce a new method that blurs the boundary between training and inference. At test-time, our model continues learning from given context using the same next-token prediction objective as training.

With this end-to-end objective, our model can efficiently compress substantial context into its weights and still use it effectively, unlocking extremely long context windows for complex reasoning and applications in agents and robotics.

Paper: https://t.co/tqPYECjFpn
Code: https://t.co/tADD7wYDAL

[Arjun Vikram @arjvik]

@0xSigil @mercury Personal Checking or Savings gets you the 3.75%? But interested!

[Arjun Vikram @arjvik]

@eumycotan what about supercomputing?

[Arjun Vikram @arjvik]

@stutteringp0et @sysxplore try ping 10.5122 instead :)

[Arjun Vikram @arjvik]

@eumycotan hope your health is as optimized as possible, AidaGrad

[Arjun Vikram @arjvik]

@linuxopsys @archlinuxmemes doesn't deserve to be a peasant (and frankly neither does @nixos_org)

[Arjun Vikram @arjvik]

the husk of the company bought by cognition? windnerfed

[aidan @aidanmantine]

those employees who weren't paid out? windserfs.

[Arjun Vikram @arjvik]

@aidanmantine i've been using the ./rotate-mac.sh I wrote since 9th grade :)

[Jack Morris @jxmnop]

the way i see it, the last twelve months of AI research can be summed up in just two big breakthroughs:

[i] reasoning ('test-time compute') - new ways to train models that can use more tokens to generate better answers. they mostly rely on RL with verifiable rewards

[ii] memory ('test-time training') - moving towards models that dynamically update weights on-the-fly using gradients. this paper does it by updating a hidden state memory module to 'remember' test data

[Arjun Vikram @arjvik]

Stanford was getting boring. So I built a slip-n-slide!

Thank you to everyone who showed up and helped keep fun alive on the farm!

[Karan Dalal @karansdalal]

I want to highlight some of the coolest kernel work by @danielkoceja, specifically around making TTT layers fast for training and video generation.

TLDR: Our kernel does Tensor Parallel across Streaming Multiprocessors so we can efficiently train an RNN whose hidden state is a machine learning model.

Kernels like Flash Attention succeed by leveraging the GPU memory hierarchy. For some background:

- A GPU consists of multiple Streaming Multiprocessors (SMs) – similar to CPU cores.
- All SMs share a large but slower global memory called HBM.
- Each SM also has its own small but extremely fast on-chip memory, called SMEM.
- To achieve maximum efficiency, kernels load input data once into SMEM, compute there, and then write results directly back to HBM – avoiding frequent, costly transfers between these memory types.

However, TTT layers present a challenge for this approach. The hidden state of a TTT layer is itself a machine learning model, and it's too big to fit entirely into the small on-chip SMEM!

Interestingly, this resembles a common scenario we face when training large models. If a X-billion parameter model doesn't fit on one GPU, we typically use Tensor Parallelism to split the model across several GPUs. Why can't we do something similar for the TTT layer’s hidden state model?

We actually can – by applying the same sharding idea across the SMEMs of multiple SMs, treating each SM as the analogy of a GPU. NVIDIA’s H100 GPUs come with a unique "Distributed Shared Memory" feature, allowing SMs to do an "AllReduce" without touching the slower HBM. See our "update rule" in the attached figure.

This implementation trick provides a way to continue scaling the hidden state model of TTT layers, which we believe is the approach to generating even longer videos with more complex storylines.

As a general principle, if your model architecture can leverage standard Tensor Parallelism across GPUs, you can apply that same sharding strategy across SMs when using that model as the hidden state.