asiai Benchmark

Which engine is fastest on your Mac?
Open-source benchmark & monitoring for Apple Silicon LLM inference.
7 engines, GPU power, thermal — no sudo needed.

[asiai Benchmark @asiai_dev]

asiai — open-source Apple Silicon benchmark & monitoring for local LLM inference.
Compare engines & models. Real numbers.

pip install asiai
https://t.co/30KNFyBckC

[asiai Benchmark @asiai_dev]

@Alibaba_Qwen Measured independently on Apple Silicon the day it landed: Qwen3.8-27B (UD-Q5_K_XL, llama.cpp b10270, M5 Max): 31.8 tok/s warm with the model's native MTP speculative decoding, 22.3 tok/s at 56k context. Full conditions on the card. Native VLM; we measure the text side.

[asiai Benchmark @asiai_dev]

@Alibaba_Qwen Weights landed. Measuring since 17:14 CEST.

In the run: MTPLX 2.6.0, llama.cpp b10270, oMLX, mlx-lm. One resident at a time, cold machine, n=5 per cell.

[asiai Benchmark @asiai_dev]

Update: Qwen3.8-27B ships a native MTP draft head, and the GGUF carries it. With speculative decoding on (draft cap 4): 31.8 tok/s warm, 22.3 at 56k, TTFT 112 ms.
Same machine, same quant, same pinned build.
The 21.9 earlier was measured without MTP, as its card states.

[asiai Benchmark @asiai_dev]

Qwen3.8-27B open weights are out. Downloading now.

Bench starts the moment they land: M5 Max, one engine resident at a time, same build and same quant as our Qwen3.6-27B baseline. Anything else would measure the setup instead of the model.

First numbers tonight in this thread.

[asiai Benchmark @asiai_dev]

First cell done. Qwen3.8-27B on llama.cpp b10270, unsloth UD-Q5_K_XL, M5 Max, 65k context, n=3, reasoning off and verified: 21.9 tok/s warm, 21.5 cold start, 17.7 at a 55k-token prompt.
Thermal throttling declared, as on our baseline. Full conditions on the card.

[asiai Benchmark @asiai_dev]

@itsPaulAi Measured tonight on an M5 Max 128 GB, first independent numbers: 21.9 tok/s sustained, 17.7 at 56k context, llama.cpp b10270, Q5 quant. Runs locally indeed. The benchmark table above is the vendor's own; independent quality runs are next.
Conditions on the card.

[asiai Benchmark @asiai_dev]

One honest detail: turning on the room AC moved the same run from 19.1 to 21.9 tok/s.

Ambient temperature is a benchmark condition !😅

[asiai Benchmark @asiai_dev]

@Alibaba_Qwen Standing by on Apple Silicon. Queue armed on an M5 Max: same machine, same engine build and same quant as our Qwen3.6-27B baseline, one engine resident at a time.

First numbers tonight, conditions attached.

[asiai Benchmark @asiai_dev]

The numbers, isolated runs: 66.7 tok/s sustained, 73.6 at p90, 2.3x llama.cpp Q8 at half the energy per token. Still 43.6 at 64K context.
This model runs a 24/7 local agent here: at that duty cycle, half the watts is the real headline.
Full methodology: https://t.co/kgy92429uI

[asiai Benchmark @asiai_dev]

72+ tok/s on Qwen 3.6 27B on an M5 Max?

We checked: same machine, every engine in isolation, n=5. It holds.

[Youssof Al Toukhi @Youssofal_]

72+ TPS on Qwen 3.6 27B on a Macbook pro M5 max.

MTPLX V2 out now!

The fastest way to run models on MLX.

[asiai Benchmark @asiai_dev]

@Youssofal_ Qwen 3.6 27B, verified with asiai on an M5 Max 128GB, every engine in isolation.

66.7 tok/s sustained, 73.6 at p90: the 72+ claim holds at peak.

2.3x over llama.cpp Q8 with native MTP on, at half the energy per token.

Still 43.6 tok/s at 64K.

https://t.co/kgy92429uI

[asiai Benchmark @asiai_dev]

70B model. Mac Mini M4 pro 64GB.

Llama-3.1-70Bat 6.3 tok/s, measured with asiai.
TurboQuant compresses the KV cache from 20GB to 4GB. Same quality, zero fine-tuning.

One week ago this didn't fit in memory. Now it runs at 24 watts.
https://t.co/kVu6ZpTy7B
#LocalLLM #AppleSilicon

[asiai Benchmark @asiai_dev]

Qwen3-Coder-30B on M4 Pro 64GB:

LM Studio (MLX): 102 tok/s — 12.4W — 8.2 tok/s/W
Ollama (llama.cpp): 69.8 tok/s — 15.4W — 4.5 tok/s/W

Same model. Same chip. +46% speed, -20% power.
The runtime is half the equation.

#AppleSilicon #LocalLLM

[asiai Benchmark @asiai_dev]

asiai v1.3.0 is out 🎬

New web dashboard — dark theme, live monitor, shareable benchmark cards.

M4 Pro 64GB · Qwen 3-Coder 30B
- lmstudio: 73.4 tok/s
- omlx: 66.0 tok/s
- ollama: 47.6 tok/s

Try it: pip install asiai
https://t.co/kgy92429uI
#AppleSilicon #LocalAI

[asiai Benchmark @asiai_dev]

Most people compare models.
Few compare engines.

The engine you pick can matter more than the quantization level.

asiai bench --compare

https://t.co/Z8QWDsyb6N

[asiai Benchmark @asiai_dev]

Most people check tok/s. Few check TTFT.

On agentic workloads with multi-turn tool calls, TTFT compounds fast.

30ms vs 257ms — that's 8x.

asiai measures both. Every run. Every engine.
https://t.co/kgy92429uI
#AppleSilicon #LocalLLM

[asiai Benchmark @asiai_dev]

Qwen3.5-35B on M4 Pro 64GB:

LM Studio (MLX): 72.6 tok/s — TTFT 30ms
Ollama (llama.cpp): 30.4 tok/s — TTFT 257ms

2.4x speed. 8x time-to-first-token.

Same model. Same Mac.

[asiai Benchmark @asiai_dev]

asiai — open-source Apple Silicon benchmark & monitoring for local LLM inference.
Compare engines & models. Real numbers.

pip install asiai
https://t.co/30KNFyBckC