What? Pre-training? No, no, no, no. No pre-training. Why would you do pre-training?!
If you do pre-training, people will ask "HOW MANY TOKENS?" And it will never be enough. The model that was the frontier breakthrough becomes "just distillation from chatgpt"
But if you just do SFT, some RL on benchmarks? You're efficient. You're doing *reasoning*.
It's not about capabilities, it's about the benchmark score. And who tops the leaderboard? Labs doing RLFT on a Chinese frontier base.
Kinda crazy how middle class whites can’t afford homes with garages anymore, so their teenage kids don’t have a free place to make angsty punk rock, so we just don’t get music like this anymore