EmbeddedLLM

Your open-source AI ally. We are committed to making production-grade AI inference as accessible and reliable as electricity, powered by vLLM.

[EmbeddedLLM @EmbeddedLLM]

Tomorrow at #SFTechWeek with @tensorwave, @CosmicLabsTech, @weka and @makora. Hardware, watts, kernel and token.
Silicon doesn't serve itself. That's the layer we run.
10am, SF. If you're shipping tokens, meet our team.

[Miguel Saavedra @miguelsaavedra]

Our @Techweek_ Inference at SCALE panel is growing!!

Excited to have our friends from @makora, @weka, and @EmbeddedLLM joining our @tensorwave x @CosmicLabsTech teams for a conversation around hardware, inference, market dynamics and more.

https://t.co/9T8Y7SHXSP

[vLLM @vllm_project]

🎉 MiniMax M3.1 Flash from @MiniMax_AI has Day-0 support in vLLM!

Built for multi-step automation, real-codebase engineering and visual reasoning—with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack.

What the model can do, and the vLLM engineering that makes it practical to serve. 👇

Model: https://t.co/M67ZM3a2EA

1/5

[Emad Barsoum @EmadBarsoumPi]

Outstanding work by the team!!! Optimizing World Action Model on AMD GPU by combining kernel level optimization via GEAK (https://t.co/hwuNtpHIwP), fewer denoising steps and compact action expert to reduce the latency from 222.55ms to 8.83ms. @AIatAMD

Fully open
📝 Blog: https://t.co/zvGCtlSsWR
💻 Code: https://t.co/TIg830S1Uh
🤗 Checkpoint: https://t.co/2Ku157bFay

[PyTorch @PyTorch]

FlyDSL, an open source Python kernel DSL, now plugs into PyTorch on @AMD MI350-series GPUs via torch.compile and eager mode.

Up to 1.68x geomean scaled GEMM kernel speedups and up to 3.01x end-to-end #vLLM serving speedups.

Read more: https://t.co/ALYzbUkdAs

[vLLM @vllm_project]

High throughput on the grill. Zero latency on the drinks.

vLLM maintainers and friends gathered during the PyTorch Conference for an evening of Brazilian steak, drinks, and great conversation! 🥩🥂

Grateful for this community and everyone who keeps vLLM moving forward. 🚀

[Woosuk Kwon @woosuk_k]

vLLM is all you need to reproduce DeepSeek's performance.

[EmbeddedLLM @EmbeddedLLM]

@simon_mo_ @vllm_project @SemiAnalysis_ @inferact Proud to work with @simon_mo_ @inferact on this!

[EmbeddedLLM @EmbeddedLLM]

@SelfHostedSI @vllm_project @AMD @Kimi_Moonshot @SemiAnalysis_ Both used official Kimi K3 MXFP4 model.
NVIDIA submitted this run. They already use NVFP4 for Kimi K2.6 on AgentX, so Id expect them to use it for Kimi K3 too if it ran faster.
BTW, SGLang tested both formats on B300. NVFP4 wasn’t consistently faster.

[EmbeddedLLM @EmbeddedLLM]

We helped close the performance gap.

@SemiAnalysis_ Kimi K3 estimator puts MI355X at 36% higher revenue per chip-hour than HGX B300, with a 38.8% profit margin vs 7.7%.

Performance = profit.

[vLLM @vllm_project]

🚀 @AMD MI355X just showed competitive performance to NVIDIA B300 on @Kimi_Moonshot K3 serving!

8× MI355X delivers 24.4% more throughput than HGX B300 (7,567 vs 6,083 tok/s/GPU) at 50 tok/s/user on @SemiAnalysis_ AgentX agentic coding benchmark, and 87.4% more tokens per dollar.

We cover the vLLM optimizations, AMD kernels, and hardware details that closed the gap, plus patterns for spotting bottlenecks so you can get more out of vLLM on your own GPUs.

Thanks @AMD @embeddedLLM @SemiAnalysis_

Full writeup: 🔗 https://t.co/jP1Dme7dPn

[vLLM @vllm_project]

🎉 Step 5 Preview has Day-0 support in vLLM.
Sparse attention, MoE, multi-token prediction and vision, with serving work across Blackwell and Hopper.
The work spans the sparse indexer, FP8 MegaMoE and cache transfer. Here is how the pieces fit together, with kernel results and validation below. 🧵
1/6

[vLLM @vllm_project]

🎉 Day-0 support for Ling-3.1-flash is here in vLLM!
~560B total parameters, ~25B active per token, and up to 1M context. From a rough brief to websites, mobile UIs, and interactive prototypes—serve it on your own infrastructure with vLLM.
Congrats to @AntLingAGI!

[EmbeddedLLM @EmbeddedLLM]

Building the future of AI on @AMD takes the whole stack working in lockstep.

From @tensorwave's Instinct GPU clusters and @weka's data layer to our work at @EmbeddedLLM pushing vLLM on ROCm and TokenVisor into production, the shift from raw GPU hours to governed, high-throughput inference and agent services is happening right now.

Great sharing the stage with @CosmicLabsTech, @makora_ai, @miguelsaavedra, and Prithvi Mattur during #SFTechWeek! 🚀

[TensorWave @tensorwave]

Thank you to everyone who came out for our “From Training to Inference” panel with @CosmicLabsTech, @makora_ai, @EmbeddedLLM, and @weka during #SFTechWeek.

It was a packed room and a great conversation about where AI compute is heading, what builders need now, and how the infrastructure behind it all is evolving.

Shoutout to our speakers @miguelsaavedra and Prithvi Mattur!

[SemiAnalysis @SemiAnalysis_]

ALERT🚨: AMD JUST LANDED SUPPORT FOR MEGAKERNEL DECODE IN UPSTREAM VLLM WELL BEFORE NVIDIA. This marks one of the first times that a major vLLM feature is only supported on ROCm and not on CUDA yet.

MegaKernels is a technique that is massively important for high-interactivity decoding by reducing the # of kernel launches by 10x.

This marks an❗️important milestone❗️for ROCm vLLM to land major ROCm-exclusive features. CUDA vLLM doesn't even have draft PRs for upstream vLLM megakernel support yet.

[Lisa Su @LisaSu]

It was an incredible honor to receive the National Medal of Science from @POTUS today.

My parents came to America with big dreams and a deep belief in education and hard work. This country gave me the opportunity to pursue my passion for semiconductors and work with extraordinary people to push the boundaries of what is possible.

I am so excited about what we can accomplish together as America continues to lead the world in science and technology.

[EmbeddedLLM @EmbeddedLLM]

Meet us at #COLM2026 in San Francisco.
Oct 9 at DAIH: our research on deploying clinical AI where the data lives across Nvidia and AMD GPU, with energy and cost in view.

Building models or putting them into production? Come say Hi! @manpree59181175
DM us to meet.

[Ramine Roane @roaner]

AMD MI355X posts the best margin on every model in SemiAnalysis InferenceX agentic profit estimates:

Kimi K3: 45.7% vs 43.8% (GB300)
GLM 5.3: 63.4% vs 63.2% (GB200)
MiniMax M3: 51.5% vs 48.4% (B300)
DeepSeek V4 Pro: 87.9% (no green data)
https://t.co/xkqMPjR1X2

[EmbeddedLLM @EmbeddedLLM]

Massive respect to @AMD, @roaner and @AnushElangovan for investing where it counts: dedicated hardware, CI, and the room for core maintainers @Rxday000 to push ROCm to its limit.

The partnership is in a better place than it has ever been. 502 commits is the baseline. When software and silicon move at the same speed, the whole open source ecosystem wins. Proud of what we’re building together on @vllm_project.

[Ramine Roane @roaner]

#1 company contributing code to @vllm_project right now?
@AMD: 502 commits to core vLLM in 90 days. 13% of all org contributions, almost 4x Nvidia.

Also grateful to Red Hat, IBM, Embedded LLM & Inferact for building it with us.

Open source wins when hardware has a choice.

[RL-Align @RLKernel]

Zero mismatches. Bit-for-bit aligned. 🔥
Incredibly proud of our team's work integrating with vime @vllm_project. Rock-solid RL pipelines are now a reality on @AMD ROCm.

Read the deep dive below and star our repo⭐ https://t.co/0euenP62l5

[vLLM @vllm_project]

Training and rollout logprobs matched bit for bit on ROCm. The @RLKernel team integrated RL-Align/RL-Kernel with vllm-project/vime. A 200-step Qwen3-8B GRPO run on 8× @AMD MI300X recorded zero logprob mismatches between Megatron training and vLLM rollout.

The strict path aligns reduction order, intermediate precision, rounding points, and math primitives across both sides.

Deep dive:
https://t.co/FVJHoyoo3t

[SemiAnalysis @SemiAnalysis_]

Massive shoutout to TJ & Team at @EmbeddedLLM; they are the lead maintainers & CODEOWNERS of vLLM ROCm! Up until May 2026, they did not even have persistent access to an MI355X cluster, but after SemiAnalysis worked with & convinced AMD leadership of something that is quite obvious in hindsight, they finally got a persistent cluster to make MI355X great on AMD!

Super proud of the @EmbeddedLLM Team, highly respected within the @vllm_project community.