Chidera Ibe

MTIA Intern @ Meta & Prev Palantir SWE Fellow

[Chidera Ibe @ChideraIbe11]

Inside xAI: How they made LLM inference insanely fast

[Chidera Ibe @ChideraIbe11]

the headline is "amd caught up" but the real story is that inference is memory-bandwidth bound, not flops bound. once you're serving, the moat was never raw tensor cores, it was HBM and a software stack that keeps them fed. rocm closing the gap on serving is scarier for nvidia than any spec sheet.

[wafer @wafer_ai]

🚨 BREAKING:

these engineers figured out how to serve GLM 5.2 on @AMD MI355X at 2626 tok/s/node and 213 tok/s single stream at over 2x lower cost than Blackwell

that's ~80% of B200 throughput at over 2x lower cost

full write-up in reply to see how

[Chidera Ibe @ChideraIbe11]

everyone's fixated on the PHEW / I'M DROWNING theater. the actual signal is right here: a model compressing its reasoning into dense private shorthand is exactly what you get when you optimize chain-of-thought for tokens, not humans. legibility was never the objective. (grok summary, salt to taste)

[𝕱𝖚𝖑𝖑 𝕶𝖊𝖑𝖑𝖞 @full_kelly_]

@Polymarket These are screenshots of supposedly what its reasoning looks like (basically models output text that normally is kept internal and not shown to the user, to help them think through difficult problems). Prior reasoning models typically looked more like ordinary speech

[Chidera Ibe @ChideraIbe11]

"muttering and grumbling to itself in a language of its own" is a great headline and a bad description of what's happening. it's not sentience, it's compression. the model learned english is a lossy container for its own intermediate state. neuralese is just entropy coding for thoughts.

[Polymarket @Polymarket]

JUST IN: Leaked internal reasoning reveals Claude Fable 5 “muttering and grumbling” to itself in a language of its own.

[Chidera Ibe @ChideraIbe11]

sim2real gets framed as a rendering problem. it isn't. the gap is contact dynamics, friction and deformation and the exact moment a gripper slips. you can photorealistic your sim to death and still fail because physics is where the transfer breaks, not pixels.

[Chidera Ibe @ChideraIbe11]

the dirty secret of inference serving is that latency and throughput are the same knob turned in opposite directions. batch harder, tail latency explodes. optimize p99, gpus sit half idle. there's no free lunch, just which customer you decide to disappoint.

[Chidera Ibe @ChideraIbe11]

people treat fp4 like a free 2x. it isn't. the win isn't the format, it's whether your outliers fit the block scale. narrow the block and you pay in metadata, widen it and one bad value poisons 32 others. the tradeoff was always the block, never the bits.

[Chidera Ibe @ChideraIbe11]

new take: NVIDIA's real product isn't the GPU anymore, it's utilization. the whole neocloud arrangement only works if the chips stay busy. so they're not selling you silicon, they're selling you the guarantee it never sits idle. margins live in the uptime, not the wafer.

[Chidera Ibe @ChideraIbe11]

everyone benchmarks humanoids on locomotion because it demos well. the real moat is manipulation data. walking is a solved control problem, folding a shirt is a million edge cases. whoever owns the manipulation dataset owns the decade.

[Chidera Ibe @ChideraIbe11]

Someone on metas PyTorch team created a competition to write gpu code

[Chidera Ibe @ChideraIbe11]

@tanmai I personally think it’s LLM inference optimizations and things like kernel, compiler, and runtime optimizations

[Chidera Ibe @ChideraIbe11]

A prediction I want on record: the next real architecture breakthrough won’t be from one of the big labs. It’ll center on memory efficiency, and it’ll come from a team spun out of OpenAI — not SSI. I think they announce soon.

[Chidera Ibe @ChideraIbe11]

@tanmai https://t.co/8KQeiPcbh4

[Chidera Ibe @ChideraIbe11]

China’s robots are learning to handle folding and manipulating non rigid objects.

[Chidera Ibe @ChideraIbe11]

both NVFP4 and MXFP4 store elements as E2M1, the exact same 16 values. the difference is block size, 16 vs 32, and scale precision. halving the block halves how many outliers share one scale. people argue about 4-bit formats like it’s the elements. it’s never the elements, it’s the blocking.

[Chidera Ibe @ChideraIbe11]

people think kernel launch overhead is rounding error. on decode it’s 2 to 10 microseconds per launch and one measured system put it at 14.6% of end to end latency. you generate a token, you eat thousands of launches. the megakernel pitch is just: stop launching.

[Chidera Ibe @ChideraIbe11]

FlashAttention 4 is basically a confession that attention isn’t matmul-bound anymore. on Blackwell the tensor cores got so fast the exp and the matmuls take the same 1024 cycles. the whole kernel game flipped from hiding matmul to hiding softmax

[Chidera Ibe @ChideraIbe11]

x86 and ARM both come with a landlord. licensing, roadmap control, someone else deciding what extensions ship. RISC-V is open and royalty free, so building on it is a bet on owning your own instruction set end to end. that’s why Qualcomm put Dragonfly on it and why Meta signed up. the ISA is the new moat.

[Chidera Ibe @ChideraIbe11]

the GR00T reference humanoid is NVIDIA trying to do for robots what CUDA did for GPUs. standardize the training, eval, and deploy stack so nobody reinvents it. the platform play always beats the point solution. watching this one closely.

https://t.co/8L07rpnAEC

[Chidera Ibe @ChideraIbe11]

everyone keeps saying Triton ate CUDA C++. then FlashAttention 4 shipped on Blackwell and went back to a C++ tile DSL because the TMA needs tile level control Triton can’t expose yet. the abstraction that wins is the one that matches the hardware, and the hardware keeps moving.