Only ~1.2B parameters active at a time. Edge0-8B-A1B-preview makes an 8B-class MoE practical for local inference.📜 Apache 2.0.
🤖 https://t.co/xJ8uHG80kB
⚡ Reaches 23.9–25.3 tokens/s with about 1.0 GiB peak active memory in the reported short-context benchmark.
🏆 Retains most of the FP16 base model’s quality, with an average gap of just 2.8 points across five benchmarks. MMLU-Pro rises from 65.8 to 70.1.
🧠 SSD expert offload keeps most weights outside active memory, while a prerouter predicts which experts each layer will need next.
🛠 Recover-LoRA offsets quality loss from 4-bit quantization and expert offloading. The current Preview runs locally through MLX on Apple Silicon.
🤯 So you can actually get a 100K context & ~50 tps on one 16GB GPU.
Putting Qwen3.8-27B onto an RTX 4070 Ti super is the kind of localmaxxing I ❤️ to hear about.
Recipe 👇
🧠 Qwen3.8-27B
🎮 RTX 4070 Ti SUPER — 16GB
📚 100,000-token context
⚡ 47–50 tok/s
💾 15.93GB VRAM
This recipe stacks several optimizations:
✧ 13.5GB hybrid GGUF
→ attention stays IQ4_XS
→ FFNs compressed to IQ3_S
✧ BeeLlama.cpp instead of stock llama.cpp
✧ KVarN compressed KV cache
→ K = kvarn5
→ V = kvarn4
✧ Last 1,024 KV tokens kept at higher precision
✧ MTP-2 speculative decoding
Changing the KV cache from kvarn5/kvarn5 → kvarn5/kvarn4 alone reportedly freed another ~6% VRAM and pushed context from 88K → 100K.
🎯 So this is what they did …
Weights → selectively quantized.
KV cache → selectively compressed.
Recent context → kept high precision.
Decode → speculated.
⚠️ This uses BeeLlama.cpp, a llama.cpp fork, not stock llama.cpp/LM Studio, and the 47–50 tps number is a community report.
Links 👇
🔗 Reddit r/LocalLLaMA/comments/1w1lq7u/qwen_38_27b_at_50_toks_with_100k_context_on_a/