@YuvrajInfer Are there papers to back this 10X claim? Otherwise this is just speculation.
Quantization in our experiments has provided about 3X kernel speedup but just about 11% gain on throughput.
FP8 KV with agentic workloads can give high throughput due to cache re-use.
We did a similar experiment - tuning for agent workloads on Qwen3.8-Flash-Next
Key differences:
SSEngine is prefill-only (returns 1 token; decode is disaggregated).
Ours is Prefill and Decode..
It runs on 4× B200/GB200 versus our 2× H200.
It uses GLM-5.3-Flash (320B total / 18B active) versus Qwen3.8-Flash-Next (~176B / 6B active).
It is measured at 400–512 concurrent agents/sessions versus the 16–32 agent points we report.
Its primary metric is request rate at extreme concurrency; we report InferenceX-style total tokens/s per GPU (cached input counted in full), TTFT, and interactivity.
On our side, at 32 agents on 2× H200 (full prefill + decode), we measured ~55k requests in a 1-hour window (~15.3 req/s). That is already in the same order of magnitude as SSEngine’s AgentX result (24.0 req/s on 4× GB200, prefill-only), despite half the GPUs, a smaller model, weaker hardware, and doing decode as well.
Our TTFT at the 16-agent point was p50 0.45 s / p90 ~0.94 s.z
Blog to be posted on https://t.co/E4zVawShH5 soon.
In an experimental single-H100 conversation-cache benchmark, LeapQuant delivered 21–31% more latency-qualified throughput than BF16-window across three paired runs, implying approximately 17–24% lower output-serving cost; answer-quality equivalence remains unqualified.
#Inference
A promising LeapQuant result: +18.4% output meeting our reported latency policy vs BF16 in one 256-user test on H100. Raw throughput rose 1.66%. This is a screening result; memory-budget tuning and quality qualification remain open.
LeapQuant benchmark update: a full 16-step decode+fold cycle against BF16 window. On one H100 PCIe at batch 512: 1.273× recurrent-layer kernel throughput.
Our implementation of LeapQuant on one H100 PCIe: our report shows 1.273× complete-cycle kernel throughput vs BF16 window at batch 512. The five-block serving pilot remains inconclusive. Next: tuned controls, repeated serving tests and quality qualification.
LeapQuant is a technique that compresses the memory linear-attention AI models carry between tokens, aiming to reduce GPU memory use and speed up inference while preserving answer quality.
Looking for fast inference?
~3× kernel speedup and ~45% more output tokens per second versus packed FP32—using training-free quantization of recurrent states in linear attention using an implementation of Leapquant
@conlesspan Yi Pan - We implemented LeapQuant (Yi Pan et al., arXiv 2609.38166), an 8-bit recurrent state for linear attention, in SGLang and measured it with Qwen3.5-9B on one GPU. At 512 concurrent requests it gave about 45% more output tokens/s than SGLang's default fp32 state,
What we are not claiming: a reproduction of the paper (its results are on other GPUs, in vLLM), production readiness, or an official benchmark result. Next: repeated paired runs, full memory accounting, and quality tests on the exact serving build. 7/7