@DGamtenadze@kaiwei_chang yes that's right, the changes were about more efficient cache management to enable higher concurrency.
these 9 iterations were all autonomous, but the loop uses my previous manual work to come up with proposed levers.
@DGamtenadze@kaiwei_chang iteration 4 was a collection of kda and mla kv cache changes - agentx is kvcache bound so these changes can really help as they let up bump concurrency up in the benchmark, which is most of the gains
not all would carry over, depends on the fix, but the loop works on any model!
We built InferenceForge 🔥, an agentic system that profiles, benchmarks, and optimizes LLM inference for agent workloads.
43% higher throughput than the best published baseline under the same latency SLA on the SemiAnalysis AgentX benchmark.
We share the lessons learned and insights on our blog, along with the code changes behind the optimizations.
🔗 https://t.co/mw8iZBPXa4
Thanks to the team! @CaiJesse, Huimin Zeng, @leoleoasd, @dakuowang, @kaiwei_chang
In just 5 days of post-training on 64 NVIDIA B300 GPUs, we built FireFlash🔥, a GLM-based long-horizon coding agent that pushes the performance–cost Pareto frontier.
🚀 46.2% on Terminal-Bench 4.0, outperforming Fable 5 Max at 30× lower cost per task.
@shreyansh_26@d_haziza@shreyansh_26 Not yet - I'm still in the process of writing some of the kernels to get e2e speedups, but looking to get something out before the SLLM workshop - will keep you posted :)
@rvarm1 For a n dimensional space the probability that a critical point is a local min/max is 1/2^n (assuming independence), as the point must be a local min/max along all dims, so saddle points are much more common