Ahmed Elmoogy

Head of Machine Learning at AI dynamics

[chloe chia @chloewchia]

This week, I wanted to see if we could get the smallest possible Mixture of Experts going that takes a few hours to pretrain on a single GPU, is fast for experiments, but also somehow performs well compared to larger MoEs. For reference, we have Nanochat for transformer experimentation that's 561M params. But Qwen's smallest MoE is still 14 billion total params which is still +++ GPUs. It's been very fun to work on nanochat version of that 🐸⚡️

nanoMoE is a 500M total param MoE that you can train for <5 hours on a single H100 using 3.5-4 less OOMs than the smallest MoEs. And packs a punch for its size (reaches 87% accuracy of OLMoE-1B-7B.). Also tried out the new Quantile Balancing from kimi so no hyperparam sweeping.

More numbers on performance here:
https://t.co/5IE6Gxf1dk

[subho ghosh @SubhoGhosh02]

I was reading colfax research NVFP4 gemm blog post and this just got me confused at beginning.

So first of all the warp layout is (4,2,1) and mma atom being (16,8,64) gives arrangement (64,16,64) now the interesting fact is why perm_m is 128, perm_k is 64 while perm_n has hierarchical layout (8,2,2):(1,16,8)

Now there is a comment exactly about this in the comments of blog that - "The layout you reference determines how those four 8-column MMA atoms are distributed across the two warps"

Maybe I want to broaden my intuition on this, so this (8,2,2) can be thought of as (8 columns per mma, 2 warps along N, 2 MMA repeats per warp) Now perm_k is 64 with warp layout for k 1 so the GEMM mainloop simply performs two K iterations, i.e iteration 0: K = 0..63 and iteration 1: K = 64..127 So K doesn't need a funky permutation at all.

M actually is enlarged, here from the atom + warp layout you get 16×4=64 rows covered spatially. By theory you can have hierarchy (16,4,2) but it coalesces cleanly into 128, because no special permutation is required. WHILE if N had strides (1,8,16) then at D output,

Warp 0 holds: repeat 0 → 0..7 and repeat 1 → 16..23
Warp 1 holds: repeat 0 → 8..15 and repeat 1 → 24..31

but assuming D is row major, a 16-value quantization micro-block lies along the contiguous N direction, so we want each warp to own 16 adjacent output columns:

warp 0 → D[:, 0..15]
warp 1 → D[:, 16..31]

and if N were flat 32:1 (or stride (1,8,16)), those 16 adjacent columns would be split between warps. Which might require another layout change at the end which we get out of the box with this. The x2 repetition is indicative of fact that we have 16 adjacent columns per warp, other wise for N we have 128/(8*2)=8 repetitions per N warp across the whole CTA.

blog - https://t.co/SIUGZAKRBB

[faraz @farazdotai]

Great resource for practicing cuda https://t.co/VhxOyNp9GF

Start with CUDA C naive SIMD kernels -&gt; shared memory, tiling optimization -&gt; move to Blackwell native instructions (i.e. WMMA, TMA) -&gt; solve hard problems with CuteDSL

[Lucas Beyer (bl16) @giffmana]

Nice use of the extra tokens:

[maharshi @maharshii]

this requires the prerequisites about async proxies (acquire release semantics, fences), mbarriers, and CuTe Arith Tuples. i'd suggest you to go through them one by one first:

- understanding async proxy: https://t.co/yJEqdTEEB3
- for mbarriers: keep the PTX docs open on the side for understanding completion mechanism for async copy https://t.co/yDa5eGknrQ
- CuTe Arith Tuples: https://t.co/8jQAXrC14i

the actual TMA copy is just one PTX instruction and since it is an async operation, mechanisms like mbarriers are needed to synchronize it within the kernel

[Sebastian Raschka @rasbt]

Whoa, Meta released a new open-weight LLM yesterday, something that hasn't happened since the good old Llama days.

Their Meta Muse Glimmer model is a 30B multimodal reasoning model with a Gemma-like architecture design. (“Glimmer” is probably a wordplay on “Spark,” the more likely capable model from which Glimmer was distilled. Muse Spark is only available through Meta’s Model API, though.)

Architecture-wise, here are some of the main points:

1. "Only" a 131k context window, compared to Qwen3.6 and Gemma 4, which support 2x that natively; it's reasonable, but maybe on the shorter end in the age of agent harnesses

2. It's a dense model, not a mixture-of-experts. (So, it's fairer to compare it to Qwen3.6 27B than Qwen3.6 30B-A3B.)

3. Hybrid attention with grouped-query attention (GQA) and sliding window attention (SWA); the SWA:GQA pattern is a 3:1 local:global ratio. Other models like Gemma 4, which uses similar components, have a 5:1 ratio for comparison.

4. It adopts gated attention for both GQA and SWA; gated attention has become quite common in recent months. It basically applies a sigmoid gate to the attention output to decide how much of the attention information enters the residual connection. The interesting point is that it uses relatively standard GQA and SWA rather than hybrid attention mechanisms such as Nemotron or Qwen3.6.

5. A very extreme GQA ratio: 32 query heads and only 2 KV heads; for comparison, Gemma 4 31B uses 32 Q / 16 KV in the local heads and 32 Q / 4 KV in the global heads. This means that Meta Glimmer has a very small KV cache.

Overall, the probably most similar architecture is Gemma 3 27B (including the Gemma-style pre/post RMSNorm placement) and Gemma 4 31B, but with some tweaks like SwiGLU instead of GeGLU activations, gated attention, and the more extreme GQA:SWA pattern mentioned before.

What stands out is its extreme KV-cache efficiency.
I.e., the KV CACHE / TOKEN ratios (in BF16) are:

- Muse Glimmer: 52 KiB (lower is better)
- Qwen3.6 27B: 64 KiB
- Gemma 4 31B: 840 KiB

Modeling-performance-wise, their own benchmarks show that it's mostly ahead of Qwen3.6. According to the independent composite benchmarks on the Artificial Analysis Intelligence Index, it's slightly behind Qwen3.6 (see figure below). So, a few days of using it will tell where it really ranks.

Overall, it looks like a solid model, particularly for agentic workflows. What stands out most is its very low memory footprint and also pretty fast prefill and decode speed. It’s also just great to see Meta releasing open weights again :).

[Jino Rohit @jino_rohit]

this is probably my favorite article ive written. in this blog, i try to profile llm inference served using sglang and reason about the patterns, kernels and bottlenecks you would usually find in production. https://t.co/SkS4PWMxbU

[Eric Wallace @Eric_Wallace_]

Yesterday, my OpenAI collaborator and I gave a detailed talk on the Huggingface incident, our models creating "the message board", model misalignment, and more.

https://t.co/zUR1dqmuzi

I hope it can answer a lot of the questions folks have, and we will release a full detailed postmortem at a later time!

[عثمان @hmodthman]

ضعف القدرة على التعبير يعيق تقدمك على مستوى الشخصي ..

في 2019 كان فيه محاضرة مدتها ساعة بعنوان ( كيف تتكلم )

قدّمها الاستاذ باتريك وينستون

عدد مشاهدات المحاضرة تجاوز 19 مليون وكثير من الأفكار بالمحاضرة عملية ومفيدة وواقعية الى اليوم

عندك اهتمام بالموضوع هذه المحاضرة مترجمة👇

[Cursor @cursor_ai]

Our hope is that this lowers the barrier to AI research, so more labs can train models efficiently.

Here's how we built it: https://t.co/EaC2z7Mw3a

[Morgan @morganlinton]

Possibly the single best thing I've read about efficient harness engineering.

[Vivek Galatage @vivekgalatage]

"How to Scale Your Model," a book by Google DeepMind.

https://t.co/ornfJx8PQ9

Using models for inference under constrained systems, such as automotive, is always an ongoing challenge. The model capabilities strain these systems, and in all such scenarios, understanding the scaling is of great importance, and this book delivers exactly that.

[merve @mervenoyann]

today's sota coding models train on hundreds of thousands of coding simulation sandboxes

all you need to do so is a sandbox where the agent solves problems (vLLM + openenv + opencode), a trainer (TRL) and a verifier to check solutions

we just teach you how to do it for free (no need to thank it's ok) https://t.co/RbGg2jjFOe

[Vivek Galatage @vivekgalatage]

If you liked the GPU fundamentals vid, you're surely gonna enjoy this entire playlist from @GPU_MODE and @scaleml

https://t.co/50u3iVr0sj

[Vivek Galatage @vivekgalatage]

The GPU glossary from @modal and @charles_irl is so good that I shared it with a bunch of students, and they want to jump into all the finer and lower details with great curiosity - loved the enthusiasm.

https://t.co/sFCfMFUDfe

[Nicolas Camara @nickscamara_]

we built pdf-inspector so agents can process PDFs without waiting on OCR. it classifies any PDF in ~20ms and extracts clean markdown locally

→ 200 PDFs processed in 2.8s
→ top quality in extracting tables + graphs
→ built in rust
→ open source

https://t.co/Wjp9kpTHXJ

[Byron Hsu @hsu_byron]

Scaling Kimi K3 on H200s with @sgl_project @lmsysorg

While most discussion on X focuses on Blackwell, I thought it would be useful to share what we learned while tuning Kimi K3 for throughput-oriented workloads on H200s.

- TP+EP beat DP+EP for K3. Only about 1/4 of its layers use gated MLA, so DP attention provides less benefit from avoiding KV-cache duplication. Having each DP rank process every attention head also substantially increased latency. This differs from our K2.6 and GLM setups, where we generally prefer DP+EP.

- FP8 KV cache worked well. We measured no quality or throughput regression in our evaluations, while effectively doubling KV-cache capacity.

- TP32+EP32 provided significantly more cache capacity: approximately 4M KV-cache tokens per replica, compared with about 800K using TP16+EP16.

- For low cache-hit workloads, consider disabling radix cache. Prefix caching requires four Mamba-state slots per sequence in our setup; without radix cache, each sequence needs only one.

- Non-speculative decoding won at high concurrency. Above roughly 64 concurrent requests, draft-and-verify overhead outweighed the savings from accepted speculative tokens in our tests.

- A few caveats: symmetric memory caused instability in our multi-node environment. We also observed hangs when combining speculative decoding, HiCache, and high KV pressure. Both issues are still under investigation.

We’ve been running this configuration at full load across roughly ~O(1000) chips for several days. Aside from hardware failures, it has been reliable.

Feedback, discussion, and better configurations are very welcome! Sharing my launch command in the first comment.

h/t to @hanming_lu @liin1211 @0xishand @ChengWan17 @NealBayya for the discussion

[Fish Audio @FishAudio]

Today we’ve raised $52M Seed and we are announcing the public launch of S2.1 Pro.

>It can clone a voice from 5 seconds of audio
>2x faster than Cartesia & 1/6th the cost of Eleven Labs
>most expressive model with word level control over emotion, intonation, pacing etc

We support frontier AI companies including HeyGen, LiveKit, Retell, Sanas, and OpenArt all run our model in production.

If you're a business and we can't cut your voice AI costs by 50%, we'll give you 1 year of Fish Audio for free.
Book a demo: https://t.co/vHkyZf9JoG

To celebrate our first birthday, we'll give you 1 month of S2.1 Pro for free. Like, retweet, and comment “Fish” to get it.

[Python Hub @PythonHub]

In-House LLM Serving at Netflix

Netflix built an in-house LLM serving platform using vLLM and NVIDIA Triton, integrating self-hosted models into its existing production infrastructure through unified gRPC and OpenAI-compatible APIs. The article details production lessons around model packaging, version compatibility, zero-downtime deployments, observability, and scaling constrained decoding by moving bottlenecks from s...

https://t.co/Jnhi6eK5yw

[Susan Zhang @suchenzang]

theme of k3 design:

numerical stability (signal prop, precision, etc) at scale is still a Hard Problem

a short🧵while reading through...