Gary Wang

[Akshay 🚀 @akshay_pachaar]

https://t.co/q4DvRIfStN

[AI Will @FinanceYF5]

前伯克利博士、现 xAI SGLang 负责人，用 23 分钟讲清楚他们如何在 10 万张 GPU 上部署 Grok

拆分 Prefill 和 Decode → 将 MoE 专家分片到不同 GPU → 按专家路由 Token → 让通信与计算重叠执行 → 以足以碾压 DeepSeek API 的价格对外提供服务。

[Avi Chawla @_avichawla]

Karpathy said something that feels wrong at first:

He said he has stopped fighting how messy agent-written code gets.

His AGENTS[.]md rules told agents to keep every line doing one thing, using intermediate variables instead of stacking calls together.

The agents kept doing it anyway, chaining functions and indexing results inline, no matter how many times he wrote the rule down.

Clearer instructions didn't solve this for Karpathy, which means the fix will lie somewhere outside prompting.

When a human opens a PR, the reviewer assumes the author understood the codebase.

An agent doesn't carry that.

It writes code that compiles and clears the obvious checks, and that's enough to land a commit.

Whatever's actually wrong with it surfaces in CI later, by which point you're already building on top of it.

Catching this before it compounds needs verification running where the code gets written, not a separate check that runs after.

Sonar Vortex code analysis engine already does this inside an agent's own session, applying the same quality and security checks your pipeline runs, regardless of how the code got written.

Gitar is Sonar's AI-native PR review layer for the pull request itself, and the team worked with me on this post to show how it works.

It reads the change with context on your codebase and conventions, not just the diff, catching functional and logic bugs a syntax-level scan wouldn't name.

When it finds something, it writes a patch, runs it against your CI, and doesn't call the job done until the build passes.

Sonar calls the full loop Agent Centric Development Cycle (AC/DC).

It covers guiding the agent, verifying what it produces, and fixing what's wrong when it isn't.

Gitar closes that loop without a human acting on a comment first.

Teams running this combination are 44% less likely to see outages tied to AI-generated code.

With Sonar Vortex, token usage drops too, by roughly 8% with up to 36%, since agents spend less time reasoning and re-parsing a codebase that isn't piling up the kind of mess Karpathy described.

The PR used to be where mistakes got caught.

Now it's where they get fixed before anyone has to read them.

Start with AI-native code validation with Sonar: https://t.co/f1cCwb4cpR.

Verification at the point code gets written is one-half of working this way. The other half is the loop behind it, why an agent can't be the one to decide it's done.

My co-founder wrote a full breakdown on that, including the maker/checker split that makes "done" provable instead of claimed.

Read it below.

[Gary Wang @garywang]

https://t.co/LKoi9LYGDz

[Gary Wang @garywang]

https://t.co/CTfdamvLmo

[Ahmad @TheAhmadOsman]

DROP EVERYTHING

The ultimate resource for running LLMs locally is now available online to read for free

Covers what to use on

- Laptop / edge / odd hardware
- Mac-first workflows
- Single RTX GPUs
- 2-4+ NVIDIA / CUDA GPUs
- General production serving
- Long-context / MoE / routing
- NVIDIA max performance
- Cluster orchestration

Software

- llama.cpp
- MLX / MLX-LM
- ExLlamaV2
- ExLlamaV3
- vLLM
- SGLang
- TensorRT-LLM
- NVIDIA Dynamo

You should read this, and if you cannot now then you most definitely wanna bookmark it for later

Opensource & Local AI FTW

[Gary Wang @garywang]

我家附近建案的建築工人每天午餐都是泡麵與一瓶茶...

[Phoronix @phoronix]

4K @ 60 FPS USB Video Capture Finally Becomes Less Problematic On Linux

https://t.co/603zbrc54V

[Gary Wang @garywang]

https://t.co/ubt9z3idsD

[Gary Wang @garywang]

https://t.co/vHeXPH9QrM

[Gary Wang @garywang]

https://t.co/3XldE4Rhjf

[Gary Wang @garywang]

https://t.co/R3yiwfDKe9

[Ronald Mannak @ronaldmannak]

Nvidia’s Local AI State of the Union. Standing room only. The interest in local AI is off the charts 🚀

[Gary Wang @garywang]

Apple Mac studio 或ultra 開始最多只到96GB memory 😮

[Gary Wang @garywang]

今天是農曆十五，吃素日

[Gary Wang @garywang]

A Japanese singer jumped into a karaoke bar in Scandinavia and sang with... https://t.co/mqn1N9VlNZ via @YouTube

[Gary Wang @garywang]

https://t.co/oim6rEFmAz

[Ahmad @TheAhmadOsman]

Local AI hardware = capacity X bandwidth X software stack

- Capacity tells you what fits
- Bandwidth tells you how hard the box can breathe
- The software stack tells you how much of the spec sheet you can actually cash out.

Hardware by Memory Bandwidth
- Mac Studio M3 Ultra: up to 512GB @ 819 GB/s
- RTX PRO 6000 Blackwell: 96GB @ 1792 GB/s
- RTX 5090: 32GB @ 1792 GB/s
- RTX 4090: 24GB @ 1008 GB/s
- RX 7900 XTX: 24GB @ 960 GB/s
- Radeon PRO W7900: 48GB @ 864 GB/s
- AMD Radeon AI PRO R9700: 32GB @ 640 GB/s
- Intel Arc Pro B65: 32GB @ ~608 GB/s
- Tenstorrent Wormhole n300: 24GB @ 576 GB/s
- Tenstorrent Blackhole p150: 32GB @ 512 GB/s + 800G
- MacBook Pro M5 Max: 460-614 GB/s
- MacBook Pro M5 Pro: 307 GB/s
- DGX Spark: 128GB @ 273 GB/s (coherent + CUDA)
- Mac mini M4 Pro: 273 GB/s
- Ryzen AI Max / Strix Halo: ~256 GB/s (~96GB usable GPU)
- MacBook Air M5: 153 GB/s
- Snapdragon X2 Elite: 152-228 GB/s
- Intel Lunar Lake: 136 GB/s
- Snapdragon X Elite: 135 GB/s
- Mac mini M4: 120 GB/s
- Arc Pro B60: 24GB @ ~456 GB/s

Verdict

- GPUs are still the bandwidth kings

- Apple wins: stupid amounts of memory, don't want to shard across GPUs
- Apple loses: when raw tokens/sec & concurrency matter more

- DGX Spark: coherent memory + NVIDIA stack

- Strix Halo / Ryzen AI Max: first real x86 unified-memory contender

- Tenstorrent: fully OSS stack, excited to see this mature

Fitting != serving

Even if it fits, you still pay for
- bandwidth during decode
- KV cache growth
- dequantization
- batching + concurrency
- scheduler quality
- framework overhead

The only mental model that matters:

1. What must fit?
2. What bandwidth tier do I need?
3. What software stack can actually deliver it?

In short:
- NVIDIA -> fastest raw speed
- Apple Studio M3 Ultra -> biggest one-box memory
- Strix Halo -> first real x86 unified
- DGX Spark -> coherent NVIDIA dev appliance
- AMD / Intel Arc -> rising alternatives
- Tenstorrent -> fully opensource stack

Do ask: "which bottleneck am I buying?"

Not: "which hardware is best?"

[Gary Wang @garywang]

AI BOX...
https://t.co/hreSyJVU0B