The GPU-hour rate leaves out the people, the setup time, and the failures.
Dec. 3, 11 a.m. PT: Surf Report and Crusoe break down what AI training infrastructure really costs over three years.
Register: https://t.co/XZP6gQGtWa
Independent testing by Surf Report. Sponsored by Crusoe.
We've been livestreaming a bit talking about AI. We'll be live again this week (link coming soon) to talk about security and all of the CVEs and other issues that seem to be nonstop now with @jgavinray. Here's last weeks stream:
https://t.co/OPSUIU1DVE
@shobhitpaliwal@CrusoeAI 4/ Slurm also runs its own GPU health checks before and after every job and while nodes sit idle. If something looks wrong, Slurm pulls the node out of rotation itself, so problems get caught even when thereβs no error signal.
@shobhitpaliwal@CrusoeAI Our Slurm team wanted me to add some additional information:
1/ Good question. On Crusoe Managed Slurm, Kubernetes manages the machines and Slurm manages the jobs. Kubernetes never kills a Slurm job on its own. It flags a problem node, and Slurm decides what to do.
@shobhitpaliwal@CrusoeAI 3/ Slinky sees that and drains the Slurm node. Slurm stops scheduling new jobs there and lets running jobs finish. Once the node is free or a 10-minute grace period has elapsed, AutoClusters swaps it out for a healthy one from a spare pool.
@shobhitpaliwal@CrusoeAI 2/ When a GPU reports an error, AutoClusters catches it right away, marks the node as unhealthy, and cordons it in Kubernetes.
My team is hiring for two roles. Both are On Site (US) - at one of our many locations. If you're interested in helping keep our team, customers, and others up to speed on what's new in AI / Cloud Infrastructure then please apply!
https://t.co/aFQVP40WhU
https://t.co/c0j1KyZpch
Fine-tuning used to mean spinning up a cluster and hoping nothing broke before the job finished. Then you got to do it again for deployment.
Crusoe Serverless Fine-Tuning and Self-Serve Deployments are now GA. Tune an open model, deploy it to production, no GPU cluster to manage.
Read more β https://t.co/O6rIJb2VbK
#FineTuning #OpenModels
Fine-tuning a model is half the job. Getting it to production without re-platforming is the other half.
Part 2 of our Serverless Fine-Tuning series: a fine-tuned Qwen3.5 9B model deploys straight to a production inference endpoint with Self-Serve Deployments. No export. No new vendor.
Technical deep dive β https://t.co/H3N7pmUWzd
#AIInfrastructure #MLOps