Holy 5h!t, it’s fast! TPU Time!
256 AI requests at once. Same hardware, same model, same precision. We're 800% FASTER!
Pretty proud of what our team is building.
*P90, Backboard vs vLLM-JAX @BackboardIO
https://t.co/11uY9YHnwZ
lol there are countless of purpose-built inference engines to bypass SGLang and vLLM’s trade-offs
everyone and their grandma is making them
most take specific high level learnings and techniques directly off of published papers and existing code and redistill into a custom implementation
this is how software is made now