Morgan A Gebhardt

Nothing to see here, move along...

[TAHO Labs @TAHOlabs]

Every GPU you pay for should go to work the moment it’s on, not wait for the next job to start.

Here a fourth machine joins an AI job that’s already running. The node count goes from three to four, the new machine takes pieces of the work still left, and nothing restarts.

[Morgan A Gebhardt @gnodis]

@TAHOlabs PS - anyone notice that they just breezed right past the fact that it's running a single inference job across a multi-node fabric? "Like, yeah...just regular day on the frontier of computer science. Distributed inference on a resilient fabric. Doesn't everyone do that?" 😆

[Morgan A Gebhardt @gnodis]

@TAHOlabs Pretty cool, if you're into that whole "jobs not failing" thing...

[TAHO Labs @TAHOlabs]

We killed a server in the middle of an AI job.Four nodes were working on one prompt. We stopped one, the node count dropped to three, and the answer kept streaming.

Nothing restarted. Nothing was rerouted.Normally the job dies with the machine and starts over. See the TAHO difference in action.

[TAHO Labs @TAHOlabs]

Day 214 of waiting on the GPU capacity request. Or you could unlock the idle compute you already have. That's what we built.