Raffaele Mancini

[ModelScope @ModelScope2022]

The next-gen architecture powering Qwen4 is now here! ✨
Get ready for the open release of Qwen3.8-Flash-Next 🚀
The countdown starts now! ⏳🔥https://t.co/0Zlesgxnot

[Raffaele Mancini @RaffoMancio]

If you want to try it on your android phone, BigMoeOnEdge supports Ling-3.0-flash and tiny, hours after llama.cpp merged the architecture upstream.

The experts stream from flash on demand, so the whole thing runs on a android 12 GB phone.
Built with llama.cpp, one registry row.

2.6 tok/s flash
16 tok/s tiny

[Raffaele Mancini @RaffoMancio]

BigMoeOnEdge support for @AntLingAGI 's Ling-3.0-flash and tiny, hours after @ggml_org llama.cpp merged the architecture upstream.
124B, 5B active.
The experts stream from flash on demand, so the whole thing runs on a 12 GB phone: 51 GB on disk at 3-bit, four times the RAM.
Built with llama.cpp, one registry row.
2.6 tok/s flash
16 tok/s tiny

[Raffaele Mancini @RaffoMancio]

@MeisterCoins @ggml_org Yeah, clearly not very usable, but it's a start. Qwen 3.6 (waiting for 3.8) runs at 7–8 tok/s

[Raffaele Mancini @RaffoMancio]

BigMoeOnEdge: streaming MoE inference on Android, built on llama.cpp (@ggml_org).

DeepSeek V4 Flash IQ2_M (92 GB) running on a 12 GB midrange phone at ~1 tok/s. Qwen3.6-35B-A3B up to 7-8 tok/s, Gemma 26B and Qwen3-30B in the same tok/s range. CPU only.

Experts stream from UFS flash with O_DIRECT reads overlapped with compute - no fork, just public APIs, bit-for-bit identical output to resident weights.

Apache-2.0, prebuilt APK, one-tap model catalog: https://t.co/U82jIyMfUu

Huge thanks to @ggerganov for ggml/llama.cpp.

[Rimsha Bhardwaj @heyrimsha]

Running a 60GB AI model on a phone with 12GB of RAM should be impossible.

Someone just did it anyway.

It's called BigMoeOnEdge and it runs gpt-oss-120b (a model 5x bigger than the phone's RAM) at 2.2 tokens per second on plain CPU. No GPU. No NPU. Four cores and flash storage.

The trick: MoE models only use a few "experts" per token. Instead of loading all 60GB into RAM, it streams only the experts each token actually needs, straight from flash, at the exact moment the model asks for them.

The rest stays on disk.

→ gpt-oss-120b (60GB) on a 12GB phone at 2.2 tok/s
→ 14x faster than the same file loaded the ordinary way (0.09 tok/s with mmap)
→ Byte-for-byte identical output to running the full model in RAM
→ Qwen3-30B at 5.2 tok/s, Gemma-4-26B at 4.1 tok/s on the same device
→ Built on stock llama.cpp, upstream untouched, tracked as a submodule
→ Android demo app with a live telemetry panel and every knob exposed

Apple, Google, and Samsung spend billions trying to fit smaller models onto their phones.

One developer streamed a bigger one from flash and got the same answer.

100% Opensource.

https://t.co/X8Qk2Y5T47

[Raffaele Mancini @RaffoMancio]

@heyrimsha Thanks a lot for the post. I’m the creator.
With the latest updates, it runs even better: Qwen 3.6 35B MoE (awaiting Qwen 3.8) at 7–8 tok/s, and DeepSeek 284B at 1 tok/s

[Raffaele Mancini @RaffoMancio]

@rusabuilds @heyrimsha Qwen 3.6 35 B now at 7-8 tok/s

[Raffaele Mancini @RaffoMancio]

@rusabuilds @heyrimsha No (I’m the creator). I never use `mmap`, and I use good quantization levels, not unusable ones (like q4, q5). The trick lies in a series of engine optimization techniques

[Raffaele Mancini @RaffoMancio]

300+ ⭐ on GitHub.

Thanks to everyone who tried it, opened issues, and sent feedback. More to come!

[Raffaele Mancini @RaffoMancio]

@paoloardoino not a phone..not for much longer..
It seems strange but ..I'm running the exact same model with the same quantization on a android mobile device with 12 GB of RAM (1 token/s ok - I'm working on optimizing it)
https://t.co/O3K1x3ksVc

[Raffaele Mancini @RaffoMancio]

@skirano

[Raffaele Mancini @RaffoMancio]

@ivanfioravanti I get it. Nowadays, everyone is an influencer, and no one wants to get their hands dirty

[OpenRouter @OpenRouter]

DeepSeek V4 Pro 0813 is live on OpenRouter.

@deepseek_ai reports large agent gains over V4 Pro Preview: DeepSWE 62.7 (+49.9), CyberGym 83.3 (+30.6), NL2Repo 61.5 (+23.0), and Terminal Bench 2.1 87.9 (+15.8)

More providers coming online soon

Use it now: https://t.co/1PYoEFbvyK

[Chris @ChrisGPT]

DeepSeek V4 Pro 0813 is out today too!

DeepSeek reports jumps from 72.1% to 87.9% on Terminal-Bench 2.1, 52.7% to 83.3% on CyberGym, and 12.8% to 62.7% on DeepSWE!!

It costs $0.435/M input and $0.87/M output with a 1M-token context window.

I don’t like how every company report very separate benchmarks every time.. So I guess we will have to rely on the index.

[Marco Franzon @mfranz_on]

Qwen3.8 2.4T on my MacBook, an honest analysis:

I have a MacBook Pro M5, 24 GB unified memory, 589 GB free disk, so a 2.4T-param MoE whose BF16 repo is 4.89 TB across 213 shards (FP8 variant: 2.5 TB) can't fit my pc.

Even a hypothetical 1-bit quant lands near ~510 GB and 95B active params means ~21 GB must be read per token, which exceeds your entire usable RAM. There is no configuration that runs this locally.

So I'll wait for the 27B version.

[Raffaele Mancini @RaffoMancio]

@shuai_bai_ Amazing thanks for the great job!

[Shuai Bai @shuai_bai_]

Qwen3.8-2.4T-A95B is out. 🚀 https://t.co/otaQYjibv6

[Hikari∣LocalLLM⚡ @Hikari_07_jp]

08-15 00:00 (UTC+09:00)
Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen 3.8 27B!!Qwen