Ravi Sharma

Food+Drinks hog..Stay weird, Stay humble! #Austin #SolutionArchitect @ #RedHat

[Yuan (Terry) Tang @TerryTangYuan]

Excited to be speaking at #SciPy2026 in Minneapolis next week!

If you're attending, I'd love to connect. Come say hi after the talk or find me during the conference. Let's chat about open source, AI infrastructure, or whatever's on your mind.

Full schedule: https://t.co/GV3AUmkeyX

[Ravi Sharma @Ravi_Sharma]

@craig_certo @cwolferesearch I actually love printing papers and make notes on them. Best distraction less medium to learn anything.

[Ravi Sharma @Ravi_Sharma]

@dbreunig @stewartbrand Can you share the source blog?

[Yuan (Terry) Tang @TerryTangYuan]

Part 2 of our 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗔𝗜 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 series is now live on Red Hat Developer: 𝘖𝘱𝘵𝘪𝘮𝘪𝘻𝘪𝘯𝘨 𝘋𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘦𝘥 𝘈𝘐 𝘐𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦: 𝘈𝘥𝘷𝘢𝘯𝘤𝘦𝘥 𝘋𝘦𝘱𝘭𝘰𝘺𝘮𝘦𝘯𝘵 𝘗𝘢𝘵𝘵𝘦𝘳𝘯𝘴.

In Part 1, we covered prefill/decode phases and the 5D parallelism framework. Part 2 dives into the three optimization levers that deliver most of the cost and latency improvements once your parallelism layout is set:

- 𝗣/𝗗 𝗗𝗶𝘀𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗶𝗼𝗻: Not a feature to toggle on - it's a deployment topology. We share how to measure whether the prefill-to-decode imbalance in your traffic justifies the split, with 25-40% cost reductions on chat and RAG workloads in our benchmarks.
- 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿e: Tiering across HBM, DRAM, and NVMe with LMCache, the difference between prefix sharing and KV reuse (they're not the same thing), and when FP8/FP4 quantization pays off.
- 𝗦𝗽𝗲𝗰𝘂𝗹𝗮𝘁𝗶𝘃𝗲 𝗗𝗲𝗰𝗼𝗱𝗶𝗻𝗴: EAGLE 3.1 now extends gains into long-context regimes with 2x longer acceptance length than EAGLE-3. But watch out - acceptance rates collapse under constrained decoding (JSON mode, tool calls), so measure before enabling on tool-calling traffic.

One insight that keeps coming up: cache-aware routing via @_llm_d_ is what turns disaggregation from a checkbox into a working system. Round-robin leaves cache hits on the table.

Co-authored with Fatih E. Nar, Yuchen Fama, and Greg Pereira. Part 3 covering deployment blueprints and troubleshooting recipes is coming soon - follow along to catch it.

Read Part 2: https://t.co/ozlVsXYUjd

[Ravi Sharma @Ravi_Sharma]

Love this recognition @robertshaw21 and @RedHat_AI..

[SemiAnalysis @SemiAnalysis_]

Around the same time, the vLLM inference engine and its underlying Paged Attention took the open-source community by storm. Started by @woosuk_k, the @vllm_project has become one of the most widely used inference engines. @simon_mo_, @kaichaoyou and @rogerw0108 from Inferact, along with @robertshaw21 and @mgoin_ from Red Hat, have been key maintainers who continue to push the project and community forward. We are deeply grateful to the Inferact and Red Hat teams. (8/8)

[Richelle🧬 @Richelle_Ji]

Do you play an instrument or sing and wanna attend @aiDotEngineer!

We’re looking for people to jam during the conference — and your talent can unlock free access to part of it.

Instruments provided:
2 acoustic guitars
1 electric guitar
1 bass
1 drum set
1 keyboard
1 ukulele
1 banjo
1 tambourine
1 pair of maracas

Share or dm me!

[Yuan (Terry) Tang @TerryTangYuan]

Excited to share Part 1 of our blog series on Red Hat Developer: 𝘋𝘦𝘴𝘪𝘨𝘯𝘪𝘯𝘨 𝘋𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘦𝘥 𝘈𝘐 𝘐𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦: 𝘊𝘰𝘳𝘦 𝘊𝘰𝘯𝘤𝘦𝘱𝘵𝘴 𝘢𝘯𝘥 𝘚𝘤𝘢𝘭𝘪𝘯𝘨 𝘋𝘪𝘮𝘦𝘯𝘴𝘪𝘰𝘯𝘴.

LLM inference is two workloads pretending to be one. The prefill phase is compute-bound, processing entire prompts in parallel to populate the KV cache. The decode phase is memory-bandwidth-bound, generating tokens one at a time. Batching strategies that optimize one phase degrade the other and and that tension shapes every architecture decision downstream.

In this post, my co-authors Fatih E. Nar, Yuchen Fama, Greg Pereira, and I break down:

- Why prefill and decode need to be understood as fundamentally different workloads
- The 5D parallelism framework (tensor, pipeline, expert, data, and context parallelism) that governs how models are distributed across GPUs
- How context parallelism is becoming unavoidable as models push past 200K+ token contexts
- Practical configuration trade-offs across different hardware budgets

Read it here: https://t.co/XEua2ABpHK

This is Part 1 of a three-part series. We'll share the remaining parts in future posts. Follow along if you're interested in the infrastructure behind serving large models at scale.

[Yuan (Terry) Tang @TerryTangYuan]

📢 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝗶𝗲𝘀: 𝗝𝘂𝗻𝗲 𝗘𝗱𝗶𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝘂𝘁!

We recently launched our newsletter publicly after sharing it internally at @RedHat_AI for over a year. The response has been incredible - we’ve gained over 𝟭𝟱𝟬𝟬 𝘀𝘂𝗯𝘀𝗰𝗿𝗶𝗯𝗲𝗿𝘀! 🎉

Our goal with this newsletter is to give a clear, community-driven view of what’s happening across the model serving ecosystem, including updates from projects like @vllm_project, KServe, @_llm_d_, @kubernetesio, and beyond.

👉 Check out the June newsletter here: https://t.co/I57MOtujZZ
👉 Subscribe to get future issues in your inbox: https://t.co/lhGwc8gJKF

🚀 Thanks to everyone who subscribed so far!

Kudos to all contributors to this edition! Eitan Geiger, Francisco Arceo, Pete Cheslock, Jooho Lee, Pierangelo Di Pilato, Ran Pollak, Nir Rozenbaum, Yuan Tang, Wentao Ye

[Francisco Javier Arceo @franciscojarceo]

Very excited to share that I will be presenting my work from @CAISconf at the World's Fair!!

Hope to see you there!

[Ravi Sharma @Ravi_Sharma]

@Stefania_druga @aiDotEngineer @SakanaAILabs Looking forward to attend your session and connect

[Ravi Sharma @Ravi_Sharma]

So looking forward to attend @aiDotEngineer in SF this year. Last year was eye opening and looking at this year’s agenda, don’t think thats going to change.

[Math Files @Math_files]

The best opening line in any textbook✍️

[Ravi Sharma @Ravi_Sharma]

@TerryTangYuan @databricks Saw this few days ago. Looking forward to try it on OpenShift.

[Yuan (Terry) Tang @TerryTangYuan]

I’ve had a great experience using Omnigent and have been collaborating with the @databricks team to land a few features and fixes (all available in release v0.2), including:

- Kubernetes & OpenShift deployment: K8s deployment manifests, an OpenShift overlay, and a UBI9-based image for RHEL/OpenShift compliance are available.
- NVIDIA OpenShell sandbox: Integrated the OpenShell sandbox launcher for running agents in sandboxes.
- Claude on Vertex AI auto-detection: Auto-detects Claude on Vertex AI via GCP ADC environment variables.
- Podman support: Added Podman as an alternative container runtime.

Great initiative! Kudos to @matei_zaharia, @alighodsi, @dennylee, and others who've worked hard on this.

[Red Hat AI @RedHat_AI]

Multi-agent systems have a hard security problem: Agent A holds a bearer token and passes work to Agent D. Agent D was never supposed to touch patient records. But it can, because the token traveled with the request.

That's a confused deputy. And your audit logs show nothing but normal authorized activity.

Kagenti fixes this at the infrastructure level. Every request carries the full delegation chain, cryptographically signed. Policy fires on the chain, not just the token.

Watch and try it yourself (link in reply):
https://t.co/CiweUKIpHy

[Red Hat AI @RedHat_AI]

Gemma 4 Diffusion landed in vLLM last week. Day 0.

First diffusion LLM natively supported in vLLM. Instead of one token at a time, it predicts 256 tokens at once and iteratively denoises them in parallel.

Result: 1,000+ tokens per second at batch size 1 on a single H100.

Built on Model Runner V2. @googlegemma

[Glitchbyte @0xglitchbyte]

What every programmer should know about memory

Still one of the most goated reads

[Ravi Sharma @Ravi_Sharma]

@suekhim Hi, woild love to have an invite for my 10 year old. He is doing good in Math but I want him to challenge himself. Coding wont hurt either.

[Logan Thorneloe @loganthorneloe]

If you want to work as a software engineer in AI, read this.

Evals are fundamental to get agents into production. By understanding them, you’ll also better understand why AI is such a difficult engineering problem.

[Ravi Sharma @Ravi_Sharma]

@dexhorthy @swyx @vaibcode You should host one in Austin someday. Would love to have you here.