Suso Fontán

Software Architect.

Side quest: AI research & AI DevOps.

[Suso Fontán @susofontan]

@olesovhcom Entiendo ahora la realidad de todos esos ajustes de precios “por culpa de la AI”…

[Suso Fontán @susofontan]

The primary concurrency bottleneck on the M3 Ultra is compute capacity. The lack of tensor cores in the GPU creates significant computational overhead, which drastically reduces decoding throughput during the prefill phase. I managed to resolve this in a new iteration by taking advantage of the Apple Neural Engine (ANE) 😋

[Suso Fontán @susofontan]

Qwen3.8-Flash-Next running locally on a Mac Studio M3 Ultra (60-core GPU, 96 GB unified memory), served by footless, a custtom inference engine with custom Metal kernels. No MLX, no llama.cpp. The goal: real work on one 96 GB machine, several agents at once.

Quant (~180B params, ~6.7B active per token)
• 512 routed experts: QTIP-style trellis, 4.0 bpw (Hadamard rotation + LDLQ), 62 GB
• dense trunk, embeddings, LM head: 8-bit · KV cache: int8
• the 102 GB n-gram table stays on SSD, rows read per token
• model + buffers: 72 GB; the rest holds sessions and caches

Quality vs bf16
• KLD 0.007 chat / 0.003 long context · 97.4% same greedy token
• top-1 vs the hosted API: 93.1% (bf16: 93.0%)
• batching and MTP speculative decoding are bit-exact

Speed (from the video)
• decode: 74 tok/s alone; 101 / 128 / 151 tok/s total with 2 / 3 / 4 requests at once
• prefill ~900 tok/s · 262K token context

For agents
• prompt cache + prefix cache (RAM + SSD): a repeated prompt starts in ~20 ms
• 3 agents × 3 turns + a 12K-token document: later turns 84% cached, first token in 1.2 s
• prefill runs beside the others' decoding instead of blocking them
• /v1/decisions + /v1/systemone: typed questions scored at the answer, ~0.8 s

More updates soon 😋

[Suso Fontán @susofontan]

Exciting news! @Alibaba_Qwen Qwen3.8-Flash-Next running at 100% on a 96GB @Apple Mac Studio M3 Ultra (including the ANE).

The model uses a custom Trellis quantization, retaining nearly original quality. Optimized for real-world workloads and concurrency, enabling simultaneous prefill + decode (prefill on ANE, decode on GPU) while maintaining 60+ decode TPS during prefill.

Incredible to see this machine pushed to its absolute limits with such an amazing model. More updates soon!

Thanks @QwenDevs for opening this beautiful model to the world, eagerly waiting for Qwen4! 🤩

[Suso Fontán @susofontan]

@SipeedLab Im in! Thrilled and eager to unlock their full potential. 😋

[Alexey Fateev @superalesha]

A 27B model in 12 GB of VRAM. Full 262K context. 70 tok/s. And now it is UNCENSORED.

I ran it on my own RTX 3090, held to 12 GB. This is Mirai S Qwen3.8-27B, 2.4 bits per weight, an 11 GB file.

With my llama.cpp port the same card did 82K context at 38 tok/s inside 12 GB. Then @CaryPalmerr took the port and did the hard part. Same card, same file, his serve:

- context: 82K -> 262,144 with q8 KV
- decode: 38 -> 69.9 tok/s
- VRAM peak: 11.7 GB
- prefill: 1,124 tok/s

He keeps part of the KV cache in VRAM and the rest in pinned RAM, and he drafts with MTP at every depth. He checked every kernel against my fork token for token and wrote down every experiment, the failed ones too. On his real 12 GB RTX 4070 it does 76.6.

My part is small. I added abliteration. You cant edit these weights, they are compressed codes, so I measured the refusal direction on the quantized model and the engine removes it while the model runs. A 1.3 MB file and 1 flag:

- harmful prompts refused: 64 of 64 -> 0 of 64
- held-out set: 78 of 82 -> 0 of 82
- decode: 69.9 -> 69.7 tok/s

Counted by a regex, and with thinking on 1 of 24 still refuses. Deep in the context it gets slower on my card: 27.9 tok/s at 60K.

So thats 12 GB of VRAM, 262K context, 70 tok/s, abliterated. Open weights, open code, runs on your desk.

Cary did a genius amount of work and almost nobody has seen it. Go follow @CaryPalmerr right now, I mean it.

serve: https://t.co/TNmpeShz06
weights + vector: https://t.co/EOEr7bXSL1

[Suso Fontán @susofontan]

@superalesha Awesome 🚀

[Suso Fontán @susofontan]

@ThurstonPowers 😋

[Suso Fontán @susofontan]

Who said you can't run a top-tier 27B reasoning model on a $150 GPU from 2016? 🤯

It’s live: run OrcaSAQ-2-27B on an NVIDIA Tesla P100 16GB via a plug-and-play Docker image!

Huge thanks to @OrcaRouter for releasing such an incredible model.

Powered by the footless engine + custom Pascal kernels:
• ~30 tok/s decode w/ MTP speculative decoding (~20 tps standard)
• ~200 tok/s prefill | up to ~100k context
• OpenAI-compatible API + JEV-like decision endpoint
• Single-stream for now

It’s experimental! But if this gets some love, I'm going to keep pushing on multi-P100 and other GPUs like the V100 and maybe the A100.

Repo & setup: https://t.co/1tbveQXeDf

I'm going to drop some sample generations in the replies 👇

[Suso Fontán @susofontan]

@0xSero I mean hardware and kernel agnostic engine not kernels 🤦🏼‍♂️

[Suso Fontán @susofontan]

@0xSero Built this using a custom engine with hardware-agnostic kernels, paired with an ad-hoc runtime optimized for this GPU. The results are awesome 😋

[Suso Fontán @susofontan]

Who said you can't run a top-tier 27B reasoning model on a $150 GPU from 2016? 🤯

It’s live: run OrcaSAQ-2-27B on an NVIDIA Tesla P100 16GB via a plug-and-play Docker image!

Huge thanks to @OrcaRouter for releasing such an incredible model.

Powered by the footless engine + custom Pascal kernels:
• ~30 tok/s decode w/ MTP speculative decoding (~20 tps standard)
• ~200 tok/s prefill | up to ~100k context
• OpenAI-compatible API + JEV-like decision endpoint
• Single-stream for now

It’s experimental! But if this gets some love, I'm going to keep pushing on multi-P100 and other GPUs like the V100 and maybe the A100.

Repo & setup: https://t.co/1tbveQXeDf

I'm going to drop some sample generations in the replies 👇

[Suso Fontán @susofontan]

Implemented batching support for footless-orcasaq-2-27B-nvidia-p100. Benchmarks show 40 tok/s at concurrency = 4 running OrcaSAQ-2-27B (@OrcaRouter) on a single NVIDIA Tesla P100.

More updates to follow.

• Repo: https://t.co/tCNprAlfTo

• Demo (pure prose):

[Suso Fontán @susofontan]

@ELPilotAI @OrcaRouter Awesome, give it a try! 😋

[Suso Fontán @susofontan]

Who said you can't run a top-tier 27B reasoning model on a $150 GPU from 2016? 🤯

It’s live: run OrcaSAQ-2-27B on an NVIDIA Tesla P100 16GB via a plug-and-play Docker image!

Huge thanks to @OrcaRouter for releasing such an incredible model.

Powered by the footless engine + custom Pascal kernels:
• ~30 tok/s decode w/ MTP speculative decoding (~20 tps standard)
• ~200 tok/s prefill | up to ~100k context
• OpenAI-compatible API + JEV-like decision endpoint
• Single-stream for now

It’s experimental! But if this gets some love, I'm going to keep pushing on multi-P100 and other GPUs like the V100 and maybe the A100.

Repo & setup: https://t.co/1tbveQXeDf

I'm going to drop some sample generations in the replies 👇

[Suso Fontán @susofontan]

@MiaAI_lab Hey! I'm bringing back past glories and building a runtime and kernel agnostic inference engine. Your work has been a huge inspiration for me, thanks!

[Suso Fontán @susofontan]

@OrcaRouter As promised. https://t.co/tCNprAlfTo

[Suso Fontán @susofontan]

@OrcaRouter Generation example -> Markdown notes app, 1 single prompt, no corrections, using next.js:

[Suso Fontán @susofontan]

@OrcaRouter Generation example -> Flappy birds 3d, 1 single prompt, no corrections:

[Suso Fontán @susofontan]

Generation example -> Pelican test, 2 prompts, first for base second for animation:

[Suso Fontán @susofontan]

Generation example -> Flappy birds clone, 1 single prompt, no corrections:

[Suso Fontán @susofontan]

Good news! OrcaSAQ-2-27B is reaching usable speeds on an Nvidia P100 🤩

• Decode: 24+ tps
• Prefill: 200+ tps on long inputs

Remember, P100 is a non tensors 2016 GPU 🤯

Wrapping up a few loose ends, organizing the project, and writing documentation now. Opening to the public soon!