your GPU dashboard says 100%. This does not mean your GPU is full.
i run GPU clusters to do LLM inference, and this metric drives more confusion than any other
#GPU#Kubernetes#MLOps#NVIDIA#infrastructure#inference
UltraBERT is here (along with super interesting lessons learned from training it)!
152M + 380M encoders, 32K context, built for classification, NER, QA, reward modeling + retrieval.
Most interesting part is the training recipe:
-5% masking for classification
-50% masking for retrieval
-Separate LM head for each
-Distillation only on the 5% sequences
Distillation findings are also really insightful. They trained a 1.3B teacher, pruned it by width to 380M and 150M, and distilled both smaller models from the teacher. But they only applied logit distillation to the 5% masked sequences.
Some other interesting training lessons:
-Encoder performance saturated early, even as loss kept falling
-Multi-stage data mixtures had little effect after fine-tuning
-32K context added during only the final 100B of 2T training tokens
-Distillation not automatically useful across every capability
Strong size-to-performance tradeoff too:
-380M leads the authorsβ comparison on GLUE + BEIR
-150M matches RoBERTa-large on classification at 2.3x smaller
-~8 point lead on code + math retrieval
-99%+ Passkey recall through 32K
Great open source release! Long live BERT βοΈ
@huggingface weights: https://t.co/fff2oCzPwo
Paper: https://t.co/QSc2COtSik
Same model, same question, same meaning, but a longer answer can improve the score ~2x. A large share of bits-per-byte (BPB), a common pre-training eval metric, reflects answer length rather than predictive quality. We propose an alternative, PriorBPB. π§΅
We were able to fine tune an open weight model (Qwen3.5-9B) on proprietary internal data at @coinbase, and it out performed frontier models, at much lower cost and speed, for a specialized task (fraud prevention in this case).
I think we'll see more of this pattern inside companies. The frontier models are amazing generalists. But increasingly you'll see specialist models out perform on more narrow tasks, either working on teams, or by themselves.
We're making it easy to fine tune models internally for our employees. We even have a skill for it in our internal harness that makes it easy to spin these up and start training them:
"/fine-tune-and-serve-llm I want to fine-tune Qwen for payments risk review. help me get started"
I imagine we'll have some backend services that could be replaced entirely by this new architecture.