Nilabhra Roy Chowdhury

I train LLMs

[Activate @ActivateSignal]

The best conversations happen off the main stage. Access to the right mentor at the right time changes everything.

💡Mentorship sessions powered by Activate in full swing at Mumbai Tech Week - with @sharad_sanghi of @Neysa and @khemaniv from @Quantiphi giving their time to the next generation of builders. This is what the ecosystem is about. 🇮🇳 #MumbaiTechWeek

[Nilabhra Roy Chowdhury @nilabhraroy]

@soumithchintala Nice!

[Sayak Paul @RisingSayak]

The @bfl_ml team released Klein KV and showed how KV-caching can incorporated in a flow pipeline 🤯

The idea is simple and elegant.

In the first denoising step, reference image tokens are included in the full DiT forward pass. Their per-layer KVs are computed and cached.

In the subsequent steps, KVs for only noisy latents are computed while the cached reference KVs are injected during computing attention.

As a result, it delivers upto 2.5x speedups for multi-reference editing tasks over Klein.

I basically learned about it from this PR:
https://t.co/4jbAboaStf

The PR is a poetry in motion and is from the BFL team itself! Kudos to them for always being the best when it comes to designing codebases for flow and diffusion models. The best!

Check out the model here:
https://t.co/f3NOHkg2HQ

[Joël Niklaus @joelniklaus]

We just released pre-mixed, pre-shuffled pretraining datasets at 100BT scale.

@asankhaya tested 50+ different mixture strategies at 1B scale. The winner? A static 50% finePDFs + 30% DCLM + 20% FineWeb-Edu blend. No fancy curriculum needed.

We scaled this up to 100BT and pre-shuffled everything so you don't have to burn compute on sampling. Just use it:

from datasets import load_dataset
ds = load_dataset("HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT-shuffled")

Browse the full smol-data collection: https://t.co/ZNLionGk1a

Reproduce it yourself: https://t.co/Y6UCzCLaOw

Read the methodology: https://t.co/Ed7ZDAQVlu

[Nilabhra Roy Chowdhury @nilabhraroy]

@RisingSayak @cneuralnetwork Why am I being tagged? 🤣

[Delip Rao e/σ @deliprao]

Adversarial attacks on vision language action models. https://t.co/auWICLcpZ3

[Atli Kosson @AtliKosson]

The Maximal Update Parameterization (µP) allows LR transfer from small to large models, saving costly tuning. But why is independent weight decay (IWD) essential for it to work?

We find µP stabilizes early training (like an LR warmup), but IWD takes over in the long term! 🧵

[Aayush Karan @aakaran31]

We found a new way to get language models to reason. 🤯

No RL, no training, no verifiers, no prompting. ❌

With better sampling, base models can achieve single-shot reasoning on par with (or better than!) GRPO while avoiding its characteristic loss in generation diversity.

[Nilabhra Roy Chowdhury @nilabhraroy]

@RisingSayak Well said. It provides insights to model optimisation after the prototyping phase and the insights help the user discover new avenues for optimisation which otherwise would have been quite opaque.

[Nilabhra Roy Chowdhury @nilabhraroy]

@giffmana 😆

[Nscale @nscale]

We’re building Stargate Norway to support the most demanding AI workloads in the world — built for scale, speed, and sustainability. @nvidia's leadership in accelerated computing makes it possible to push the limits of what’s technically achievable, from training massive foundation models to deploying ultra-low-latency inference applications.

🎥 Watch the full announcement: https://t.co/Q3FJRbahv8

[Nilabhra Roy Chowdhury @nilabhraroy]

@Swatch Got mine in The Mall of Emirates :)

[Nscale @nscale]

GPT-OSS-120B and GPT-OSS-20B are now live on Nscale as day-zero serverless endpoints.
No orchestration required. Just build.

You’ll also find Nscale listed as an inference provider on @huggingface, making it even easier to get started wherever you build.

At Nscale, we’re committed to getting powerful AI into the hands of practitioners—quickly, safely and without lock-in.

Give them a whirl: https://t.co/xG02OEmG7D

[Nscale @nscale]

Today, we’re proud to announce Stargate Norway—a landmark initiative by @nscale, in partnership with Aker ASA and @OpenAI — one of the most significant AI infrastructure investments in Europe.

[Nilabhra Roy Chowdhury @nilabhraroy]

@tenderizzation 😆

[Nilabhra Roy Chowdhury @nilabhraroy]

@rohanpaul_ai Thanks for the share!

[Nilabhra Roy Chowdhury @nilabhraroy]

@jxmnop What a legend.

[Sayak Paul @RisingSayak]

Want to benefit from `torch.compile()` while hotswapping LoRA adapters into your diffusion models?

This is now possible, thanks to the OG @BenjaminBossan's incredible hard work!

Follow the comments for a tutorial, code, etc.

[Nilabhra Roy Chowdhury @nilabhraroy]

@cheatyyyy @JeffDean @jeremyphoward You don’t need all the experts to be in memory all the time. Experts can be swapped in and out as needed. This is will be slow but possible.