Billy Dickson

Training LLMs

[jianlin.su @Jianlin_S]

A Brief Look at K3's MoE and Attention
https://t.co/LIF6OnT7wV

[Charlie Marsh @charliermarsh]

Instead of building FlashAttention or Transformer Engine from source, you can point to our GPU indexes:

• Like PyTorch, one index per CUDA version (e.g., https://t.co/dTYOpk5Usn for CUDA 12.8)

• Each wheel is tagged by the CUDA and PyTorch version (e.g., +cu.12.8.torch.2.10)

[jianlin.su @Jianlin_S]

Deconstructing Scaling Laws: The Triad of Optimization, Architecture, and Data
https://t.co/jbqiFXsIJN

[Nathan Lambert @natolambert]

My book, Reinforcement Learning from Human Feedback is done!

This is the book I wish I had when learning to fine-tune, align, & now post-train models since ChatGPT. The resource has been built by me finding time to study and document the fundamentals on nights and weekends since 2024.

Transferring as much of the intuitions of building Olmo as I possibly can in the book format.

The book is launching with an over 10 hour, full course with slidedecks, functional code for the training chapters, an example model completions library, and of course the free online web version.

Physical orders from Manning will ship in 1-2 weeks, and Amazon a week or so after. Thanks for your support!

[Cerebras @cerebras]

https://t.co/FvKy1StDpQ

[Kimi.ai @Kimi_Moonshot]

Introducing Kimi K3: Open Frontier Intelligence

🔹 2.8 Trillion Parameters, 1 Million Context, Native Multimodal
🔹 Kimi Delta Attention enables up to 6.3x faster decoding in million-token contexts
🔹 Attention Residuals deliver ~25% higher training efficiency at <2% additional cost
🔹 Built for long-horizon agentic coding and self-evolving workflows

Kimi K3 is now live on on https://t.co/zrk6zZxZUo, Kimi Work, Kimi Code, and the Kimi API.
Open Weights by July 27, 2026.

🔗 API: https://t.co/XCrgjXAqMw
🔗 Tech blog: https://t.co/YTfiMSNM1f

[Lilian Weng @lilianweng]

new post on harness engineering for AI self-improvement: https://t.co/ZYvGfVs61k

It is hard to forecast how much the future of RSI will rely on harnesses. Likely harness engineering will evolve in the direction of self-improvement and enable auto-research, and, in turn, smarter models keeps harnesses simple.

Even when many harness improvement get eventually internalized into core model, the need to specify goals and context will not disappear.

[Amanda Askell @AmandaAskell]

@deepfates Come up with candidates, then imagine you're the pokemon version of the thing being named and say the name over and over again if it was your only means of communicating with the world. Time to despair is probably correlated with name quality.

[Alex Hägele @haeggee]

Our paper is now on arXiv: https://t.co/lfRLCyLrCx
Besides all the details and discussions of the broader literature, it also contains lots other experiments that answer some of the questions we have already received. For example:

[William Merrill @lambdaviking]

New work from @YanhongLi2062 digging into the specific tokens that hybrid models predict better than transformers 📈

Spoiler alert: gains are broad across token categories, especially large on content words. Gains diminish on copying tokens, but even there hybrids aren't worse

[Lilian Weng @lilianweng]

A super long overdue (3+ years?) post on scaling laws.

Compute is expensive. Scaling laws are a way to help us reason about the optimal compute allocation between data and model size before committing to a large run.

The post covers what scaling laws predict, how compute-optimal allocation works, why Kaplan et al. and Chinchilla disagree, and how data limits + fitting details make extrapolation tricky.

https://t.co/HP26eJvjHB

[Jack Morris @jxmnop]

https://t.co/w8pDpes7zL

[Harold Benoit @harold_matmul]

MFU is nice because it allows us to summarize a training run "efficiency" with one number. However it is not sufficient at the very large scale :)

Indeed, what labs optimize for when launching a training run is roughly approximated by
~ (final checkpoint performance after RL) / (# GPU hours.)

There's many parts of a lab recipe that will influence this.

For example, a less MFU-friendly (e.g. more sparse) model might be more data efficient, and thus the overall training time will be lower.
Your global batch size (gbsz) and the number of chips used in a training run will also greatly influence the achievable MFU. This is also coupled with your optimizer choice (AdamW, Muon, etc..).

Additionally, co-designing your model to be inference-friendly may be more important than MFU, if you spend lots of compute on RL.

[Goodfire @GoodfireAI]

#4: fart fishing

Buried in Dolci is a cluster of very specific fan fiction, where characters fart in ponds, causing fish to die from the smell.

The chosen responses in the dataset wrote vivid scenes, while the rejected refused, teaching the model to comply! (7/9)

[Dwarkesh Patel @dwarkesh_sp]

Recently met @srush_nlp and he started giving me an impromptu lecture on how targeted on-policy self-distillation works.

I asked him if I could record it on my iPhone.

The basic idea is this: if the model made a mistake at some point in the rollout (for example, calling a tool that doesn't exist), we want to discourage this specific error, but we don't want to just learn from the final reward, because it's a very noisy signal spread out over the whole trajectory.

So we have another model read this trajectory and figure where the error was made. It simply inserts some hint tokens to the part of the trajectory right above where the mistake was made.

Now with these injected hint tokens, have the model run a forward pass. You're not having to regenerate a new rollout - aka no new decode required.

The hint causes the model to assign lower probabilities to the error tokens. You then trains the original model to match these new probabilities, teaching it to downweight that specific mistake.

[Billy Dickson @billy_d]

@soldni how much hill climbing can be attributed to Talking Rain AQA Max™ Wild Berry Zero Sugar + Natural Caffeine

[Luca Soldaini 🎀 @soldni]

Climbing with no distillation, like the Big Boys do, has been super fun! Read the tech report for a taste of our ̶s̶u̶f̶f̶e̶r̶i̶n̶g̶ journey

[Prime Intellect @PrimeIntellect]

Introducing Renderers

RL trainers work in tokens. Environments work in messages. Going back and forth corrupts sampled tokens, wasting compute on every agentic turn.

With Renderers, we fix this mismatch. This unlocks &gt;3x throughput on popular open models.

[Thinking Machines @thinkymachines]

With the model's simultaneous speech capability, Horace has gotten a lot easier to work with recently.

[SemiAnalysis @SemiAnalysis_]

For the past 12 years, cuDNN has been completely closed sourced (besides the .h files), until this week! OVER 20 MoE kernels & NSA sparse attention kernels from cuDNN has been open sourced! Great work to @manicely6005 & the rest of the team on seeing that parts of NVIDIA are moving towards open kernels! open source kernels drive innovation! (1/3) 🧵