Manan

AI Research Engineer | MLSA |
Exploring LLMs, Multimodal Vision & Generative Models
GitHub:

[Manan @_manan2005]

That might mean that there is still room to RL kimi k3!

[Together AI @togethercompute]

.@Kimi_Moonshot K3 lands on Together on Monday!

We ran 452 DeepSWE rollouts against Claude Fable 5: near-flagship coding at ~35% of the price, and K3 pulls ahead at higher pass@k's.

Full deep-dive: https://t.co/9mmP6pHTKu

[Manan @_manan2005]

@varunneal @jxbz Awesome blog! shouldn't we bring back shared expert here? might help with validation loss? As all experts can learn a mutual orthogonal direction which helps them all?

[Manan @_manan2005]

https://t.co/NcU7wvv01s

[Manan @_manan2005]

This paper sparked my interest in linear attention and I have been learning about how it evolved, over the past few days. This is far from complete and I wish to do some research experiments around the same! Below is the link for the github repository.

[Rosinality @rosinality]

https://t.co/2vyOwdBVgE

Product Key Memory + Gated DeltaNet.

[Manan @_manan2005]

This is a really interesting direction. This can help with model merging/continual learning if can be adapted for transformers though closed form solution for rms norm would be a little tricky.

[Hossein Mobahi @TheGradient]

Excited to share about what I’ve been working on over the past year: quantifying the capacity of a neuron😅

This led to a mathematical framework we call HOPE, which lets us rigorously deconstruct what deep networks might have learned. Paper: https://t.co/bDn03zBkS0

[Manan @_manan2005]

Felt like the world started with compute poor suddenly we had a lot of compute credits to be won in several ways and cheap compute now we are back to being compute poor and not so cheap compute.

[Manan @_manan2005]

@jasoncwarner Would love for the base model!

[Manan @_manan2005]

@kaikuspa @MiaAI_lab @poolsideai cause of looping problem shared in discord server

[Manan @_manan2005]

This is amazing would love for someone to reduce the reasoning length for deepseek/glm models

[Richard Chen @richardczl]

Reasoning models have a verbosity tax, and it turns out you can distill it away without touching the reward function.

In our new Miles blog, we ran pure On-Policy Distillation on Qwen3.5-35B-A3B: zero task reward, only reverse-KL against an RLVR teacher that had learned to reason ~3x shorter.

What surprised me is how fast it worked. Rollout length collapsed from ~18.6k tokens to 5.5k–6.7k after the FIRST update — and held-out DAPO accuracy improved, 0.8457 → 0.8945. The student didn't trade accuracy for brevity; it just stopped wasting tokens.

Under the hood, the part I'd highlight is sparse per-position teacher scoring. The naive path materialized an O(R²K) JSON payload for a computation that only needs O(RK) values — on long-response workloads, teacher scoring drowns in payload before it drowns in compute. Sparse scoring is what makes this practical at throughput.

Thanks @nvidia — single 8×B200 node, fully colocated. 👇

[krishna @OccupyingM]

since the last one blew up here's a video for the below problem too.

this time we actually got the chat thread so i was able to get fable to try to show the entire process and thinking that was used here

[Manan @_manan2005]

Really interesting, the paper shows that, under same compute budget, starting RL from a stronger pretrained model win. But I'm curious about can a weakly pretrained model ever catch up with enough RL? If yes, how much additional RL compute does it need? or is it capped?

[Pavel Izmailov @Pavel_Izmailov]

New paper: Understanding Reasoning from Pretraining to Post-Training!

We study the full LLM training pipeline from pretraining to post-training, find a joint scaling law, figure out how the compute should be allocated, and study what RL is doing to the policy.

🧵

[Manan @_manan2005]

Such a cool implementation!

[Marcel Rød @marcelroed]

Introducing the world's fastest tokenizer implementation, Gigatoken!

Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.

These baselines are already multithreaded Rust implementations! 🧵

[Manan @_manan2005]

@ar0cket1 tpus bake architecture in the chip? what?

[Manan @_manan2005]

@ar0cket1 Hey man wanted to dm you about something but your dms are blocked. Would really appreciate if you could open them?

[Manan @_manan2005]

@ar0cket1 how about decoupling facts and capabilities and developing continual learning separately for each?

[Manan @_manan2005]

@AroraBhavyam me

[Manan @_manan2005]

@ar0cket1 same here man :(

[Manan @_manan2005]

@auto_grad_ pretty cool!