AMD AI Guy

Running models you never thought AMD silicon could run
MI-Instinct evangelist | EPYC abuser
AI • GPUs • CPUs • open weights

[AMD AI Guy @AMD_AI_Guy]

Beware of accounts posting performance numbers without saying upfront which quant they're using

[AMD AI Guy @AMD_AI_Guy]

Different design point on the same DeepSeek-V4.1-Flash: no GPU, no SSD streaming. pure CPU (48 threads, Q4 GGUF, entirely held in RAM.

Custom kernels hitting 22.5 tok/s. Effective memory bandwidth ~165 GB/s vs 460 GB/s ideal.
Plenty more work to do!

https://t.co/PFp6R8ftKA

[Kyle Hessling @KyleHessling1]

Absolutely incredible!

Two changes to default @coldniko Strata settings and I’m at 40tps median on Ds4.1 flash with the incredible IQ2_XXS from @mhelcig on a single 5090, with 128gb of throttled ddr5!

1. A bigger RAM store for the experts:
--ssd-cache 104
This did almost all of it: 21 tps jumped to 39 tps. The model has 15,360 experts. 2,680 fit on the 5090, and the other 12,680 (about 109 GB) are computed by the CPU. The CPU reads them through a RAM store that bypasses the normal file cache, and my Strata install had sized that store at 48 GB. Every answer re-read about 65 GB from the SSD. Raising the store to 104 GB keeps nearly all of them in RAM, which cut SSD reads to about 3 GB per answer and still leaves about 26 GB of RAM free.

2. A smaller prompt chunk:
--prefill-chunk 4096

Default was 8192. Changing increased prompt processing speed significantly. At about 7.8K tokens, prompt reading went from 543 to 724 tok/s, and 2048 was worse (658).

Kept the drafter off because it takes too much VRAM in my case with one GPU. Also had to mess with the cpu threads, 18 threads outperformed 20 on the intel ultra 7 265k!

Speed gradually declines as context fills, but is still very good for long context chatting. 20tps is the floor for a completely full 250k context chatting.

Just incredible performance for a 763B model on one GPU; now time for some test cases! I did not think this model would ever be usable on these specs!

[AMD AI Guy @AMD_AI_Guy]

This is exactly what I've been building with

https://t.co/QWOtA5AerO

Keep your largest/slowest model (on CPU+RAM in my case) for concise guidance and consultations, never for actual implementations

[Jason - looking for work @foley2k2]

@0xSero I'd gladly take 20t/sec if my hardware could get that out of a 1T model without a subscription. The planning would go there and the implementation would swap in a 200 t/sec model. Review and critique would go back to the big one.

[Mia @MiaAI_lab]

NVIDIA released an interesting new FP4 quant of Qwen3.8 Flash Next. Per its model card, it loses almost nothing vs BF16.

I'll check this one out.

https://t.co/IUC3cX47TJ

[AMD AI Guy @AMD_AI_Guy]

Remember to tell your GPUs that you love them ❤️

[SSH @SSHCodes]

Let me show you something crazy.

This is Qwen 3.8 27B, and every frame is 1 real equation from the model's actual math. The prompt is easy and thinking is off: we ask what the capital of France is, with a one-word reply.

If you think this video is doing math fast, it's actually skipping about 7.2 MILLION equations between every frame. After all the math, Qwen gives the final answer: "Paris".

That one-word answer took 25.8 BILLION equations, and my NVIDIA RTX PRO 6000 Blackwell did it in about 13 milliseconds. This video is 32 billion times slower than the actual GPU. Showing every single equation at one per frame at 60 fps would take over 13 years, for one word.

Now imagine scaling that from 27B parameters up to models with hundreds of billions or even trillions of parameters, and that's why AI needs so much compute.

Appreciate your GPUs, no matter which one you have. They're actually insane.

[AMD AI Guy @AMD_AI_Guy]

You can get 64GB of vram with 1.6 TB/s bandwidth for $4k

[Alexey Fateev @superalesha]

Micron already sold 75% of its 2027 memory.

Data centers bought the RAM you needed to run a model at home. Now they rent that model back to you by the month.

You can already see it in prices. DGX Spark launched at $3,999 and the same 128GB box is $6,950 now. Nvidia says the reason is memory supply. Micro Center wants your ID to sell you 1 RTX 5090.

Every year local AI needs less hardware because the models get smaller and smarter. And every year that hardware gets harder to buy. I dont think thats a coincidence I should wait out.

My 4x 3090 cost me about $6K. I couldnt build the same rig for that money today, and next year it will be worse.

Buy your hardware while you still can!!! I am not selling my 3090s.

[AMD AI Guy @AMD_AI_Guy]

I couldn't agree more with this. Stop obsessing on the hardware everyone craves. Look at the value is the specs under the hood. The kernels and quants that use that hardware at CUDA-level efficiencies are one autoresearch agent loop away. Stop wasting money on NVIDIA

[FHILY👑 @Oluwaphilemon1]

Qwen3.8-27B is hitting 273 tok/s decode on a custom LLM engine.

And that’s probably more interesting than the number itself.

The model is running with the TQ3_4S-v2 quantization, but the real work is happening underneath it.

This is a custom inference engine being built specifically to squeeze more performance out of local models.

For context, the same Qwen3.8-27B TQ3_4S-v2 ecosystem has already shown how much the serving stack can change performance. A published llama.cpp recipe reaches around 125 tok/s on an RTX 3090 using DFlash2, while the model itself is only around 13.8GB at roughly 4.24 BPW.

Now we’re seeing a reported 273 tok/s from a completely different engine.

That’s more than a small tuning improvement.

It suggests the next gains in local AI aren’t necessarily going to come from making the model smaller.

They can come from rebuilding the machinery that runs it.

And apparently that’s exactly what’s being worked on.

The upcoming announcements include:

A new quantization format

Not just another standard GGUF conversion, but a new way of representing the model specifically around the performance and memory constraints of local inference.

A new web LLM

Which sounds like the goal is to make this technology accessible directly through a browser rather than requiring everyone to understand command-line inference stacks, quant formats and GPU backends.

A smaller model for a 16GB RTX 5060 Ti

This one is particularly interesting.

The RTX 5060 Ti has 16GB of VRAM, and we’ve already seen community work get Qwen3.8-27B itself onto that class of hardware with aggressive quantization and MTP.

So a purpose-built smaller model could push the experience considerably further.

Instead of forcing a giant model into limited VRAM, the idea becomes designing the model around the hardware from the beginning.

And then there’s the biggest announcement:

A new LLM engine.

That’s the piece I’m most curious about.

Because we’ve seen the same basic phenomenon repeatedly with Qwen3.8-27B.

Different quantizations.

Different speculative decoding systems.

Different KV-cache formats.

Different kernels.

Different runtimes.

Completely different throughput.

The official Qwen3.8-27B stack already exposes things like its built-in MTP head, 262K native context and multiple speculative-decoding paths.

Community engines are taking that even further.

One runtime might give you 50 tok/s.

Another gets 100.

A heavily tuned speculative setup can go much higher.

And now a custom engine is reporting 273 tok/s.

That’s why I don’t think we should look at local AI performance as:

Model ��� GPU → tokens/sec

It’s becoming more like:

Model → quantization → memory layout → kernels → speculative decoding → KV management → runtime → hardware

Every layer can change the final result.

And that makes custom engines increasingly important.

We’re reaching a point where writing a better inference engine can produce a bigger practical performance jump than buying a faster GPU.

The really interesting question now isn’t just:

“How fast is Qwen3.8-27B?”

It’s:

“How fast can someone make Qwen3.8-27B run if they redesign the entire inference stack around it?”

273 tok/s is already a wild number.

But with a new quantization format, a new engine, a browser-based LLM and a smaller model targeting 16GB GPUs all coming next, this feels less like a single benchmark and more like the beginning of a much bigger local AI stack.

[AMD AI Guy @AMD_AI_Guy]

Another CPU data point: I'm seeing ~26 tok/s single-stream decode on DeepSeek-V4.1-Flash using EPYC 9655 + 529 GB DDR5 (no GPU). Q4_K/Q8 mixed quant + DSpark speculative decoding, measured via llama-server on my experimental development champion.

https://t.co/ZXbNAF81MV

[0xSero @0xSero]

Deepseek-v4.1-Flash on CPU + 1x 3090 Updates!

- 262k kv cache
- 600-800 tok/s prefill
- c1 now 26 tok/s decode
- c4 now 41 tok/s decode

You'll need:

- 1x 24gb vram gpu
- 256gb ddr4/ddr5
- 200gb NVMe

https://t.co/KjQv4PlLyB

[AMD AI Guy @AMD_AI_Guy]

@sudoingX Not having gone dual cpu

[AMD AI Guy @AMD_AI_Guy]

And I'm here holding the fort on the old mi210 Models

[AI at AMD @AIatAMD]

Open collaboration is moving GPU kernel development forward.

AMD and contributors from @Caltech, @Stanford, @togethercompute and the broader developer community teamed up to bring HipKittens support to AMD Instinct MI455X GPUs. The collaboration produced peak-performance GEMM kernels for the Helios GPU architecture using HipKittens.

The walkthrough starts with a baseline kernel and progresses through optimizations including asynchronous memory transfers, Tensor Data Movers (TDM) and workgroup-cluster multicast. Developers can follow the code and see how each technique shapes kernel performance.

Explore the technical breakdown and code: https://t.co/BQ6FKvesHV

[AMD AI Guy @AMD_AI_Guy]

🫠

[AMD @AMD]

This is what it takes to run agentic AI at scale.

Helios delivers over 18,000 CDNA 5 GPU compute units, 4,600 Zen 6 CPU cores and 31 TB of HBM4 memory.

All in a single rack. Looking back at #AdvancingAI with Dr. Lisa Su.

[AMD AI Guy @AMD_AI_Guy]

standardized results pages coming soon

[AMD AI Guy @AMD_AI_Guy]

If you build your own kernels you can save 5-10x on hardware costs right now. Beat the crowd before it makes you pay for understanding it later

https://t.co/tfQVOOlTuh

[Ahmad @TheAhmadOsman]

Most notable thing

M5 Ultra with 1.2 TB/s bandwidth is only 6.4% faster in token generation than the DGX Spark (273 GB/s)

M5 Ultra has ~5x the Bandwidth of the DGX Spark

[AMD AI Guy @AMD_AI_Guy]

Now picture what this looks like on an MI210

[0xSero @0xSero]

Claude took an Intel GPU, and built kernels that got within 8% of the memory bandwidth ceiling

555 / 608 GB/s

Incredible technology, this is so interesting.

[Sudo su @sudoingX]

meanwhile there are people arguing about whether one used rtx 3090 is worth it.

the men at the top are hoarding compute like it's the only thing that will matter. buy a gpu. buy the cheapest one you can find. just get on the board.

[AMD AI Guy @AMD_AI_Guy]

Pretty soon they'll start streaming weight updates through RSS feeds. lol

[Christian Elton @christianelton]

The gap between major model releases keeps shrinking.

2023: once every 73 days
2026 (so far): once every 18 days 🤯

[AMD AI Guy @AMD_AI_Guy]

Been saying this for months. Stop fixating on the hardware everybody wants to use out-of-the-box. Put a little bit of effort getting your kernels just right. There is a ton of cheap hardware out there that is severely underpriced for what it can do! @AIatAMD

[0xSero @0xSero]

At this point Claude/GLM/Kimi can write kernels making any quant run on any hardware

They can systematically discover and measure why kernels are slow.

They can manage fleets to fix every single thing physically possible.

Any hardware which can run something, will and fast.

[AMD AI Guy @AMD_AI_Guy]

😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍

[AMD @AMD]

Bringing AI into the enterprise doesn’t have to mean starting from scratch.

AMD Instinct MI350P is designed to bring LLM-scale AI into existing environments with an air-cooled form factor that helps organizations deploy without rethinking their entire infrastructure.