i generally don't think there's much to "learn"
> get the GPUs (the hardest part)
> set up vllm or sglang (they have recipes)
> set up a few replicas on the same node
> set up a cache aware router (the one from sglang should do)
> set up a cache like lmcache so your replicas can see cache hits when they are under load
> set up a gateway to manage replicas that are far apart because you run on heterogenous compute
> enjoy
see? very straightforward
Everybody I see learning Inference Engineering.
Inference Engineering might be one of the most important jobs in AI right now.
One 2026 industry analysis reported Inference Engineer job postings up 163% YoY.
And current roles with disclosed salaries average around $250K/year, with frontier labs going much higher.
If you're working in AI and not learning inference engineering yet, this is a good time to start.
You can literally get a production model trained in less than 24hrs and less than $1000 in most cases, especially if your provider has thought about cost and speed (like us!)
Come make production models with us, https://t.co/C4JRPj9oFB
This simple idea (use the largest model at the highest reasoning when developing, once done - build the specialized targeted model for production) is gradually percolating through the industry. I am surprised the adoption is not higher, though - constantly see people doing literally the reverse...
@karmakarthik not really -- they transferred much of the speedup to SGLang
and, if you're a proper provider dynamic in flight batching is almost always required
rl gets 20% because rollouts are forward-only decode, which is bandwidth bound, and 4-bit weights are half the bytes. pretraining never gets that discount
Introducing Cloudy, a research tool for long-running experiments in the cloud.
Your experiments move between CPUs and GPUs as needed, with terabytes of storage each. Pause anytime and resume right where you left off.
https://t.co/FWb9jsb7uf