The hardest part of training a 72B model over the internet with untrusted peers isn't the optimizer or the bandwidth. It's that you can't see what anyone is doing. Every decision about the run, learning rate,
intervention timing, participation thresholds, you're making partially blind. That problem never goes away. You just get better at working around it. Having done the hands-on mining work first gave me a better sense of it.
Covenant-72B is the largest model ever pre-trained in a fully permissionless setting. And it holds up against centralized 70B models.
https://t.co/7ZVZqzGdyp | @covenant_ai