Zro

Private, high-speed, multi-region inference for coding agents. Open models. Zero request retention.

[Omer Shlomovits @OmerShlomovits]

Here are some details on the four @sgl_project PRs that have come out of this collaboration so far.

While our earlier Kimi K3 work focused on getting the model running on CDNA3, this time the focus is on performance, cost efficiency, and scale.
Full benchmarks coming in the next few days.
@vllm_project PRs coming soon!

→ MXFP4 MoE on gfx942
Adds a working MXFP4 MoE path for AMD CDNA3 (MI300X/MI325X), where AITER’s native A4W4 path currently cannot execute. The implementation routes MXFP4 expert GEMMs through AITER’s Triton tl..dot_scaled kernels, adds a fused SiTU epilogue for Kimi-K3, and provides gfx942-specific tiling that fits CDNA3’s LDS limits. It works with stock AITER and is isolated to gfx942, leaving existing MoE paths unchanged.

→ Split-KV verify optimization
Brings SGLang’s Split-KV speculative verification kernel to gfx942 and makes its parallelism CU-aware. Instead of a fixed 16-split cap, it dynamically increases the number of KV splits when small speculative batches underutilize the GPU, while preserving existing behavior for already-saturated workloads. It also adds bidirectional/non-causal draft-block support and routes DRAFT_EXTEND_V2 through the optimized path. On MI300X with a 250K-token prefix and q_len=4, latency dropped from 7.63 → 0.26 ms in BF16 and 9.30 → 0.23 ms with FP8 KV.

→ Chunked prefix KV
Enables chunked prefix-cache attention for AITER on ROCm, fixing a major long-context memory problem in MLA extend. Previously, every extend operation could re-gather and up-project the entire cached prefix through kv_b_proj; the new path processes the prefix in bounded chunks and correctly merges attention states/LSE across them. For a 937K-token Kimi-K3 prefill at TP8, transient memory dropped from 19.3 GiB → 1.99 GiB, leaving 17.9 GiB free at peak, with only ~1% additional prefill latency. Peak transient memory effectively becomes independent of total context length.

→ Zro MLA attention backend
Adds a dedicated MLA backend for gfx942 targeting Kimi-K3 decode and speculative verification using custom A16W8 kernels - BF16 queries with FP8 KV. It directly supports Kimi-K3’s TP8 geometry of 12 query heads, which AITER’s gfx942 MLA kernels otherwise cannot handle, and adds a multi-query verify kernel supporting speculative windows of 4–8 tokens, beyond AITER’s current FP8 limit of 4. Unsupported shapes transparently fall back to AITER, while graph-safe metadata handling and adaptive KV-split planning improve serving behavior across batch sizes.

Many people at @AIatAMD to thank, but I have no idea how to find them all here, so I’ll just cc @AnushElangovan :)

[Zro @zroai_]

GLM‑5.3 Flash is now live on Zro.

We’re giving it special treatment from day one, with a focus on fast, reliable serving.

More performance improvements are coming over the next few days.

[Z.ai @Zai_org]

Introducing GLM-5.3-Flash

- Leading capabilities at a highly competitive price
- Natively multimodal with a 1M-token context window
- A 320B-A18B model released under the MIT License
- Previously previewed as Ox Alpha, running entirely on Chinese AI chips

Blog: https://t.co/tzOmB7gdZP

Available now across all official platforms:

Weights: https://t.co/9LRMahY9Wa
API: https://t.co/VcaQnzYmS9
Coding Plan: https://t.co/Nk8Y98HNhU
ZCode: https://t.co/Peepqv4XSx
Chat: https://t.co/WCqWT0qCQb
AutoClaw: https://t.co/aGEG5HqTTb

[Zro @zroai_]

We’re excited to share that AMD has provided Zro with evaluation compute to accelerate Kimi K3 inference serving on AMD MI325X (CDNA3).

Our goal is to push the performance of frontier-scale LLM serving on AMD hardware, and upstream the work so the broader ecosystem can benefit.

We started by building a Kimi K3 serving baseline on @sgl_project and are now working across several layers of the stack, from MXFP4 MoE execution and attention to KV-cache efficiency and speculative decoding.

The first pieces of this work are already upstream for review in SGLang, split across 4 PRs:

→ MXFP4 MoE on gfx942
https://t.co/Ph0KXlYSeE
→ Split-KV verify optimization
https://t.co/mbpWaMG5a3
→ Chunked prefix KV
https://t.co/GuhOJeFxYM
→ Zro MLA attention backend
https://t.co/H4wQNAqmFf

More performance results to come.

Thanks @AIatAMD and @sgl_project for supporting the work.

[Zro @zroai_]

@thelaggingway As we speak..

[Zro @zroai_]

😍

[Weikeng Chen | BIP-420 🐱 @weikeng]

@eli_crypto @bridgemindai My old friend, I am here to recommend you SuperGrok Heavy and please support @zroai_

[Weikeng Chen | BIP-420 🐱 @weikeng]

@eli_crypto @bridgemindai My old friend, I am here to recommend you SuperGrok Heavy and please support @zroai_

[Omer Shlomovits @OmerShlomovits]

We implemented this banger, and it's now merged into @sgl_project main 🎉

PR: https://t.co/y6rIw8FtXB

[Zro @zroai_]

@ryanjosebrosas dmed u

[Zro @zroai_]

🚀

[Stephan @stephancill]

If opencode go's deepseek inference has also slowed down for you and you're looking for an alternative check out @zroai_ https://t.co/jSbDR6d6e2 - it's fast af

shoutout @OmerShlomovits for letting me try it out!

[Omer Shlomovits @OmerShlomovits]

Serving GLM5.2 on TPUv7 - we really like this HW.
The code we pushed is a pallas kernel for DSA prefill that takes care of sparsity. It is part of a series of PRs for LLMs building blocks on TPU.

[Zro @zroai_]

Joining the party with our first PR merged: a fused sparse-MLA prefill path for DSA.

Running GLM-5.2 on 8xTPUv7x, this reduced long-context prefill time by up to 4.05× at 48K context while maintaining accuracy.

https://t.co/XiNuThuHP8

[Google for Developers @googledevs]

Big news: @Google and @RadixArk are partnering to bring @sgl_project to Google Cloud TPUs!

✅ Run SGLang on TPU today via SGL-JAX

✅ Coming soon: SGL-torchtpu for a PyTorch-native experience

✅ No more migration tax—just ultimate flexibility for devs

Learn more 🛠️: https://t.co/occdgoODu2

#GoogleCloud #TPU #SGLang #AI #DevRel

[SGLang @sgl_project]

Kimi K3 on CDNA3 with SGLang! 🧡 Love seeing this come together.

Big thanks to @OmerShlomovits and the @zroai_ team for making it happen, and to @digitalocean and @AMD for the hardware.

[Omer Shlomovits @OmerShlomovits]

Tldr-

1. For the model we tested, Fireworks appears to use off-the-shelf @vllm_project with a relatively simple routing setup

2. As a customer, you can exploit that routing behavior to improve your own inference economics

[Omer Shlomovits @OmerShlomovits]

https://t.co/SJ1baCWpr8

[Zro @zroai_]

@thelaggingway @lukatdamovie @ProductHunt @vercel For now, we’re waiting on GLM 5.3. We’ll share an update as soon as we have more to share.

[Zro @zroai_]

Honored to be included, thank you @ProductHunt and @Vercel.

[Product Hunt 😸 @ProductHunt]

The results are in 👀

Congrats to @gthartley and the @nitrosendx team on winning the top prize on @Vercel Day 🏆

Also taking home Vercel Startup Perks: Alert Grouping by @DrDroidDev, @ParadigmStudy, In Parallel MCP, and @zroai_.

[Zro @zroai_]

A faster MLA decode kernel for Kimi-K2.7-Code on MI300X

The stock attention kernel wasn't a good fit, so we wrote our own: 1.9× to 22.9× faster across the shapes we serve.

Here's what was wrong, and what we built. https://t.co/UplO79F9lE

[Zro @zroai_]

Zro is now available as an inference route for miners on Bittensor’s GM subnet.

GM miners can source GLM-5.2 and Kimi K3 through Zro when fulfilling requests.

More open-model inference, more routes for the network.

[Zro @zroai_]

@georgepickett kindly check your dm🙂

[Zro @zroai_]

one command to launch Prime Intellect and use open-weight frontier models optimized for agentic usage.

most powerful model:
> zro launch prime --model kimi-k3

fastest model:
> zro launch prime --model deepseek-v4-flash-0731

[Prime Intellect @PrimeIntellect]

Introducing Prime Agent:

A self-improving RLM harness for coding and long-running autonomous tasks.

Designed to be both token-efficient and expressive through programmatic tool calling, context as a variable, multi-agent messaging, and a self-modifiable harness state.

[Zro @zroai_]

Zro ❤️ SGLang

[SGLang @sgl_project]

Kimi K3 on CDNA3 with SGLang! 🧡 Love seeing this come together.

Big thanks to @OmerShlomovits and the @zroai_ team for making it happen, and to @digitalocean and @AMD for the hardware.