adil

agency is all you need. RL systems @NVIDIAAI. prev: @VectorInst @UofT. views and opinions expressed are my own and don’t represent NVIDIA.

[adil @a_gi_dil]

Highly belated post but NeMo Automodel now has support for large-scale MoE training w/ EP + various optimizations (DeepEP, custom layer kernels with TransformerEngine, kernel fusion, etc)!

We are seeing great perf across the board up to >30% MFU!
https://t.co/LK1URZc2pZ

[adil @a_gi_dil]

Yep. Tweeted about this a while ago https://t.co/jpzdDHmtho but it makes total sense that not every token deserves the same amount of flops. Some token (or group of tokens) might have more importance than anothers from an information gain perspective

[Mark Saroufim @marksaroufim]

Can a model run deeper at test time than it was ever trained to? And if depth becomes a loop instead of a stack, do we need better optimizers to keep training stable?

@CoreAutoAI is collaborating with @tilderesearch on an optimizer x architecture competition: "One Layer Deeper"

We picked a cursed problem y = x^(2^T) mod N where each squaring depends on the last. A vanilla transformer has a fixed depth so it falls flat on its face the moment T exceeds the serial compute it can do in one forward pass. A fun reminder that architectures have inductive biases.

This isn't quite a nano-gpt speedrun experience and it's not a typical @GPU_MODE kernel competition either, it's closer in spirit to the LLM efficiency competition I worked on many years ago with @weiwei_msr and @Jisaacso in that it's open ended in a specific way.

The UX is kernelbot like, your submission is a single python file that defines your model, optimizer and loss. You don't need a GPU to participate at all since we're reusing a queue based job system powered by @modal and @northflank

There's tons of unexplored ideas and I'm really not sure what will win out but I'd be particularly excited to see people playing with new optimizers, adaptive depth and depth extrapolation.

Since this is a new format, we'll work with the community to refine the rules for another week before we start for real. Submissions are open now!!

[Jian Hu @hijkzzz]

1/ Still looking for a minimalist, high-performance framework for agentic RL research?

Meet Molt — an agentic-first, PyTorch-native reinforcement learning framework with roughly 9K lines of RL code for 700B models.

⭐ https://t.co/B4Kde1ihxl

[adil @a_gi_dil]

@xanderai hah, same. I'm probably downing >0.5L a day of milk of Fairlife or Lactantia (helps me hit protein goals) though I try to stay mindful of the fat content and prioritize skim alternatives. Got a little spooked when you mentioned quitting cow dairy, but thanks for clearing it up :)

[adil @a_gi_dil]

@yifanzhang_ We also have integration with many popular RL frameworks!

veRL: https://t.co/AoTifQBzmP

NeMo RL: https://t.co/whFpJSE3vB

[Yifan Zhang @yifanzhang_]

Impressive! We foresee that Automodel will become one of the leading training frameworks for AI research.

[Bernard Nguyen @bernard__nguyen]

🚀 HuggingFace Transformers v5 + NeMo AutoModel = 3x faster MoE fine-tuning — with same API.

v5 ships the MoE foundations and makes them extensible. NeMo AutoModel builds Expert Parallelism, DeepEP and TransformerEngine kernels on top:

https://t.co/lInkpRn4Hm

[Jeff Boudier 🤗 @jeffboudier]

@NVIDIAAI @huggingface Amazing work, the perf gains are brrrr 🤝

[adil @a_gi_dil]

Excited to see this blog post go out: NeMo AutoModel delivers 3.4-3.7x higher training throughput for popular MoE models while staying close to the Hugging Face workflow users know.

Really proud of what the team has achieved here 🎉

[NVIDIA AI @NVIDIAAI]

The rise of MoE models introduced new challenges in training, and @huggingface's Transformers v5 brought first-class support for solving them.

Now, NeMo AutoModel builds on top of v5. Part of the NeMo framework for building models at scale, NeMo AutoModel brings optimizations to a broad set of model families through support for Expert Parallelism, DeepEP, and TransformerEngine kernels with a few lines of code.

We found NeMo AutoModel brings a 3.4 to 3.7x higher training throughput for popular MoE models. You can read more here: https://t.co/TNlBsKWwrJ

[adil @a_gi_dil]

@YouJiacheng and NVIDIA Toronto. We have a great inference team here as well

[faraz @farazdotai]

Looking for a Fall intern to work with me on agentic kernel generation/optimization for LLM inference on TensorRT-LLM @nvidia.

Great fit if you’re into compilers, GPU/ML performance, kernels, or systems. US/Canada.

Apply by email only: [email protected]
Subject: Fall Intern Candidate - Agentic Kernel Tooling

Plz include your resume + any relevant blogs, papers, or open-source work.

[difficultyang @difficultyang]

I am sad that I will not get to use Fable to work on PyTorch. I also think that these interventions are 100% consistent with Anthropic's stated beliefs. Additionally, by being first, Anthropic gives cover to other labs to impose similar limits for their frontier models.

[adil @a_gi_dil]

@aarnphm @Modular congrats! Cool to see more companies expanding here

[adil @a_gi_dil]

@_arohan_ https://t.co/UoyT8co2Qn

Also a relevant one by the 🐐 @Guodzh and James Martens

[Geoffrey Hinton @geoffreyhinton]

I just had my first ride with HOVR, a Canadian alternative to Uber. They are cheaper and pay their drivers more. They already have over 3000 drivers in the Greater Toronto Area. You can download HOVR in App Store and Google Play using this link 📷 https://t.co/2hYMXAWGw6

[Keller Jordan @kellerjordan0]

Modded-NanoGPT Optimization Benchmark

Hundreds of neural network optimizers have been proposed in the literature, recently including dozens citing Muon: MARS, SWAN, REG, ADANA, Newton-Muon, TrasMuon, AdaMuon, HTMuon, COSMOS, Conda, ASGO, SAGE, and Magma, to name a few.

The majority of this innovation is happening in the public research community. But the community currently lacks a widely accepted, easily accessible way to compare and make sense of the deluge of methods. As a result, promising new ideas get buried, and spurious results go unchallenged.

To help address these issues, I'm releasing a new optimization benchmark. It's designed for maximum simplicity and speed: Just a single file containing ~350 lines of plain PyTorch, which can complete a baseline LM training within 20 minutes of booting up a fresh 8xH100 machine. It also works with {1,2,4}xH100 or A100. These attributes make the new benchmark more accessible than any prior work.

The rules are simple: The optimization algorithm can be changed arbitrarily, with the goal being to minimize the number of training steps needed to reach 3.28 val loss on FineWeb (this is the same target loss as in the main speedrun). Modifying the architecture or dataloader, on the other hand, is not allowed. Wallclock time is unlimited, in order to give a fair chance to optimizers which would need kernel work or larger scale to become wallclock-efficient.

Like the main NanoGPT speedrun, submissions are open, and new results will be publicly broadcast. Beyond just improving the step count record, another goal of the benchmark is to collaboratively produce well-tuned baselines for as many optimizers as possible. For example, any improvement to the benchmark's best hyperparameters for AdamW would be considered a worthwhile new result.

This benchmark is not intended to be the final measure of optimizer quality across all domains. Convenient shared experimental infrastructure which covers the full space of possibilities -- across varying batch size, tokens per parameter, model scale, epoch count, and architecture -- is desirable, but far beyond the current status quo. This benchmark is only meant to be one step towards that goal.

To start the benchmark off, I've spent ~20 runs tuning baselines for Muon and AdamW. From time to time over the next few weeks, I'll add another optimizer from the literature, with my best effort at finding good hyperparameters. Researchers interested in neural network optimization are invited to join in by picking an optimizer and giving it a try on the benchmark. All optimizers are welcome, and even runs that don't necessarily have the best hyperparameters are desirable additions to the repo, because each new run adds to the collective knowledge.

[adil @a_gi_dil]

@deltaVee42 @DavidDuvenaud @demishassabis arXiv papers aren't as OOD as, for example, general relativity :P also, cutoffs are blurry nowadays

[adil @a_gi_dil]

@DavidDuvenaud What do you think about using vintage models to test whether they can independently rediscover ideas that humans only arrived at after the model’s cutoff? I think @demishassabis mentioned something like this recently

[David Duvenaud @DavidDuvenaud]

Announcing Talkie: a new, open-weight historical LLM! We trained and finetuned a 13B model on a newly-curated dataset of only pre-1930 data. Try it below!

with @AlecRad and @status_effects 🧵

[adil @a_gi_dil]

@_arohan_ case in point: deepseek