Inkling 2 Small is our first model trained partly in NVFP4, with a mixed-precision recipe that enables a great tradeoff between speed and accuracy. Check out our new post to see how we did it, and the lessons we learned along the way!
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
https://t.co/CIMECSEpD0
Hey Albert, I want to clarify our recipe uses an RHT in forward (for a different purpose), and SR separately in backward. We don't apply any RHTs in the backward pass, or quantize the backward pass to 4 bits like you do. That being said this was a really great paper that inspired me a lot at the start of my PhD last year.
Today on Connectionism: how we trained Inkling 2 Small with a native 4-bit forward pass on @nvidia GB300s, from the first pretraining step through the final RL rollout, one of the many improvements across the stack that took pretraining from weeks to 68 hours.
Congrats @thinkymachines team on Inkling 2 Small!
A 594B model pretrained in just 68 hours with NVIDIA NVFP4 on GB300 NVL72 racks.
Read how the team made low-precision training work at this scale. 👇
Inkling 2 Small is our first model trained partly in NVFP4, with a mixed-precision recipe that enables a great tradeoff between speed and accuracy. Check out our new post to see how we did it, and the lessons we learned along the way!
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
https://t.co/CIMECSEpD0
ML researcher: ...then we must de-quantize the tensors.
Me: That's not a real thing, that's some gibberish the villain says in a cheap sci-fi movie.
ML researcher: ...
Claude: That just means storing each number with more memory bits.
Me: Oh okay, why didn't you say so.
the NVFP4 recipe behind Inkling 2 Small: 4-bit forward from the first pretraining step to the final RL rollout, MXFP8 + stochastic rounding for backward
and still more to come 🪜
When I first talked to @jackcookjack, he thought that although he was having fun working on nvfp4 pretraining in academia, he was skeptical that it would ever actually work for actual large-scale runs.
As it turns out, he was wrong :^) This blog post explains how it can be done
Our v2 small model is here.
I've been using it daily for a variety of things.
The pre-training ran for 68 hours on GB300s. Post-training was 17 million rollouts over 9 days.
Entirely pre-mid-post trained in NVFP4.
Available to train and fine-tune on Tinker right now.
Open weights next week after external safety testing is fully complete (cant skimp on that).
Continuing to decentralize and democratize intelligence...