Jun beom Lee

Co-founder @fractalyze_io. Working on inference optimization, compilers, GPU kernels, and ZK.

[Jun beom Lee @jun_beom_lee]

Nice to see vLLM pick this up.

We cut Qwen3-Omni time to first audio from 213ms to 23ms on a single RTX 5090. The more important next step is turning the reusable pieces into upstream improvements, not keeping this as a one-off optimization.

[vLLM @vllm_project]

Great work from @fractalyze_io optimizing Qwen3-Omni on vLLM-Omni for a single RTX 5090. Their AWQ-4bit, batch-1 text-prompt tests cut time to first audio from 213ms to 23ms vs. stock vLLM-Omni.
We’d love to see these optimizations contributed upstream to vLLM-Omni so more users can benefit!😄

[Jun beom Lee @jun_beom_lee]

A 2x headline is only useful with its workload behind it. The TP8, fixed-prompt, capped-output setup makes the next problem clear: search configs against the traffic an endpoint will actually serve.

[vLLM @vllm_project]

Built-in DSpark: 1.92–2.16x output throughput in FP8 and 2.00–2.23x in NVFP4, versus the matching baseline.

TP8 · 200 real prompts · greedy decoding · ≤512 output tokens.

Cross-layer KV sharing, context parallelism and NIXL P/D serving handle long requests.

[Jun beom Lee @jun_beom_lee]

I want this in south korea

[Jun beom Lee @jun_beom_lee]

Off to San Francisco! ✈️🇺🇸

I’m a co-founder of Fractalyze. We’re a small team of engineers obsessed with making AI inference faster and cheaper without sacrificing quality.

I’ll be in SF from Oct 28 to Nov 6 and would love to meet founders and engineers building with open-weight models, especially in LLMs, image, video, and voice AI.

Would love to hear what you’re building, exchange ideas, and grab coffee!

DMs are open ☕️

[Jun beom Lee @jun_beom_lee]

Multimodal workloads make input mix part of serving config: vision/video change memory, batching, and scheduling. I want the optimizer to search that per workload, then validate user-visible behavior.

[vLLM @vllm_project]

Great work from @AMD and the vLLM Semantic Router team on d3-ultra! Open multimodal decision-making with per-answer probabilities, visual grounding, and video tracking—trained and served on AMD Instinct GPUs.
Apache-2.0 weights on @huggingface.

[Jun beom Lee @jun_beom_lee]

The optimization layer I want to build should search caching, quantization, kernels, parallelism, batching, and scheduling for a particular checkpoint, GPU, and workload, then keep only configurations that preserve quality and API behavior.

[Jun beom Lee @jun_beom_lee]

One decode setting is useful, but the hard part is keeping it valid as traffic changes. I want the layer to search topology and scheduling per workload, then validate quality and API behavior.

[vLLM @vllm_project]

Scaled out with P/D disaggregation and Wide-EP on a GB300 NVL72 rack, vLLM serves V4.1-Flash on SemiAnalysis AgentX at 954K tokens/s per GPU with 196 tokens/s P90 per user, 7.9× our single-node result.

Our engineering blog covers the work behind it: https://t.co/LhhM2vdFgl

[Jun beom Lee @jun_beom_lee]

Small open checkpoints make configuration search essential. I want a serving layer that chooses quantization, batching, and cache policies per device and workload, then validates behavior users see.

[NVIDIA AI @NVIDIAAI]

Available now: NVIDIA Nemotron 3.5 Bolt E4B 4GB ⚡️

Our newest small open model is built for local assistants, interactive characters and robots that can carry on a conversation and use tools to get things done.

Watch Reachy Mini chat and move, with Bolt handling conversation and NVIDIA Cosmos handling vision. Both models run locally on Jetson Orin Nano 8GB.

Learn more: https://t.co/B8ODaWMn0O

[Jun beom Lee @jun_beom_lee]

Day-0 support turns serving optimizations into reusable choices. Search should choose KV format, decoding and parallelism per checkpoint and hardware, then validate quality and API behavior.

[vLLM @vllm_project]

🎉 MiniMax M3.1 Flash from @MiniMax_AI has Day-0 support in vLLM!

Built for multi-step automation, real-codebase engineering and visual reasoning—with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack.

What the model can do, and the vLLM engineering that makes it practical to serve. 👇

Model: https://t.co/M67ZM3a2EA

1/5

[Jun beom Lee @jun_beom_lee]

Same checkpoint is not enough when moving serving across accelerators. I want the search loop to test cache, decoding and topology per hardware target, then re-check quality and API behavior.

[Google Cloud @googlecloud]

Native PyTorch on TPUs is here—and TorchTPU is officially open source.

Train and serve models using the tools you already know: @vllm_project, @sgl_project, @_llm_d_, TorchTitan, @huggingface, and @raydistributed.

Learn more from #PyTorchConf2026 → https://t.co/mISCD65vA4

[Jun beom Lee @jun_beom_lee]

One request and 512 concurrent requests tell different stories. I want an optimizer to search decoding, batching and scheduling against production traffic mix, then re-check quality and API behavior.

[Eagle @EagleCorp]

EAGLE-4 is here.

From EAGLE-1 to EAGLE-3, we’ve kept making inference faster.

EAGLE-4 is our fastest yet.

961.6 tokens/s on @Alibaba_Qwen’s Qwen3.8-27B with TokenSpeed (BF16, 4× @NVIDIA B300, TP4, one request, T=0).

Github: https://t.co/GwocZk1UTU

[Jun beom Lee @jun_beom_lee]

The unit is the serving path, not one kernel. Sparse attention, KV format, topology and decode policy should be searched together against the workload, then checked for quality and API behavior.

[vLLM @vllm_project]

Inside the serving stack: sparse attention and native low precision.

MiniMax block sparse attention uses NVFP4 attention and indexer KV. vLLM’s Blackwell MSA kernels convert NVFP4 KV directly to FP8 with CUDA 13.4’s QMUL4 instruction, avoiding the previous FP16 round trip.

Result: up to 1.7× faster indexer QK decode and 1.3× faster sparse-attention decode at the kernel level.

NVFP4 MegaMoE also fuses the MXFP8 shared-expert path; earlier TMEM release reduces prefill kernel latency by 20% in the reported settings.

4/5

[Jun beom Lee @jun_beom_lee]

Speed is not enough if an optimization shifts the logprobs training later recomputes. I want the search loop to treat output equivalence as a hard constraint, not validation bolted on afterward.

[vLLM @vllm_project]

1/ 🚀 New on the vLLM blog: Finish RL Training Faster — Bitwise-Consistent, High-Performance, Agent-Driven

Same weights don't guarantee the same policy: inference optimizations can change the logprobs that training recomputes.

We make rollout and training bitwise identical while retaining high-performance inference paths. Here's how 🧵

https://t.co/MRwfLSFKad

[Jun beom Lee @jun_beom_lee]

Cold 1M-token prompts show why cache wins cannot stand in for long-context capacity. I want the optimizer to search the serving stack against the actual context mix, then re-run the task eval.

[Stephen J Ronan MD @SJRonanMD]

1M-token context on one DGX Station GB300: two models, measured.

We gave GLM-5.3-Flash and DeepSeek V4.1 Flash 1017 needle-in-a-haystack prompts of 125K, 250K, 500K and 1M tokens, each on a single GB300. Every prompt was read cold (0% prefix-cache hits).

Needles found (out of 6):
GLM-5.3-Flash NVFP4: 6/6 at 125K, 6/6 at 250K, 6/6 at 500K, 4/6 at 1M
DeepSeek V4.1 Flash 1017: 4/6, 4/6, 5/6, 4/6

Time to first token:
GLM: 5.0 s, 10.2 s, 21.6 s, 48.8 s
DeepSeek: 4.7 s, 9.8 s, 20.9 s, 51.2 s

Prefill speed stays above 19.8K tok/s even at 1M (GLM 25.6K to 20.7K, DeepSeek 27.0K to 19.8K).
Decoding after a 1M-token prompt: 188 tok/s on both.

The test: each length has 6 items hidden in public news text. 4 are single needles at different depths, and 2 are two-hop chains: find which vault an agent uses, then find that vault's code.

What the misses tell you:
- Every miss on both models is a two-hop chain. Single needles never missed at any length.
- GLM's two misses at 1M weren't wrong answers. Both spent the whole 1,500-token reply budget reasoning and never answered. A bigger budget might pass; we haven't tested that.
- DeepSeek's older checkpoint misses the same items, so this isn't a regression in 1017.

Setup:
- GLM on SGLang with the DFlash2 drafter, all in GB300 memory. It needs --mem-fraction-static 0.88 to fit a 1M-token KV cache (1,147,584 tokens).
- DeepSeek on vLLM with the DSpark drafter and cold experts in Grace memory.

Caveat: 6 items per length is a small sample. Treat this as a capability check, not a benchmark.

Full method, scripts and logs: https://t.co/MSbaB0U89f
@Zai_org, @deepseek_ai

[Jun beom Lee @jun_beom_lee]

Most inference tuning still ends as a pile of one-off flags. I’m building toward a layer that searches checkpoint, GPU, input shape, and serving configurations together, then validates quality and API behavior before it changes production.

[Jun beom Lee @jun_beom_lee]

The durable part is the loop: search configurations against a real workload, then re-run the eval after every change. That is how same-checkpoint optimization becomes something teams can trust.

[Fractalyze ❄️ @fractalyze_io]

1/ $4.03 → $0.03–0.07 per task.

Same model, Qwen3.8-27B, on @harvey's LAB benchmark. About 60x below the leaderboard entry, with no new model and no fine-tuning.

What we changed:

[Jun beom Lee @jun_beom_lee]

Manual inference tuning does not scale because the best configuration depends on the checkpoint, GPU, input shapes, and traffic. I want an optimizer to search the combinations and validate quality and API behavior before it changes a serving stack.

[Jun beom Lee @jun_beom_lee]

@fractalyze_io I'd like to automate finding the cheapest model + serving setup that meets your workload's quality bar. Both are open to change. 🔧

[Jun beom Lee @jun_beom_lee]

You shouldn't need an inference team to get a stack tuned to your workload.

That's what we're working toward. 🔧

[Fractalyze ❄️ @fractalyze_io]

1/ OpenRouter's cheapest price is not your lowest inference cost.

That price was set for everyone who shares the endpoint. A stack tuned to your own traffic can cost far less.

What that takes, and what we've measured:

[Jun beom Lee @jun_beom_lee]

@guifav I wonder whether carrying tokenizer state across agent turns changes this more than just parallelizing lookup: it could avoid rescanning the cached prefix altogether. Does their setup have a way to reuse token IDs across calls?

[Jun beom Lee @jun_beom_lee]

The key is that AgentX and MLPerf expose different serving paths. I want an optimizer to search against the actual interactivity target, then validate the latency and cost tradeoff, not carry a tuning across workloads unchanged.

[vLLM @vllm_project]

The numbers come from two benchmarks. On @SemiAnalysis_ AgentX, vLLM serving MiniMax M3 on Vera Rubin NVL72 delivers 7.8x+ the throughput of GB200 at matched interactivity. In MLPerf Inference v6.1, vLLM with Dynamo reached up to 3.7x the throughput of GB300 NVL72 on Qwen3-VL-235B-A22B.

🧵 2/5