Same visual quality, very different speed. Qwen 3.8 27B (Q4) did ~18–20 tok/s and needed ~2,800 tokens. Inkling 2 Mini hit 55–65 tok/s with only ~3.4B active parameters and finished with ~1,300 tokens. Easy win for local use on a MacBook.
Tested Qwen 3.8 27B vs Inkling 2 Mini on my 32GB M5 Pro MacBook using the classic “pelican riding a bicycle” SVG prompt. 🚲
Honestly, the visual quality was pretty close. Both created clean, detailed scenes. Inkling 2 Mini looked slightly sharper to me, but there wasn't a huge difference.
The real difference was speed.
Qwen 3.8 27B (Q4)
Around 18–20 tok/s
Used roughly 2,800 tokens
Took a little over 2 minutes
Inkling 2 Mini
Around 55–65 tok/s
Activates only ~3.4B parameters per token
Used roughly 1,300 tokens
Finished in about 22 seconds
So, nearly the same visual quality, but Inkling was much faster and used fewer tokens.
For local use on my MacBook, that's a pretty easy choice. Inkling 2 Mini is staying.
New paper shows 25× lower end-to-end latency on generative recommendation inference vs vLLM — without writing a single custom kernel.
They used Tree Attention to share the prompt prefix across beams (instead of treating beams as independent sequences), static KV cache allocation, pre-generated masks, and graph compilation.
Result: generation latency matched a SOTA custom-kernel solution.
Main takeaway: most of the win came from fixing redundant KV reads and host-side overhead in the serving pipeline, not from faster attention math.
If your bottleneck is still “we need a better kernel,” check whether the pipeline itself is re-reading or re-launching work first.
(https://t.co/1E19PT66VO)
We measured 47 custom attention kernels on H100.
Only 6 of them improved end-to-end decode latency by more than 10%.
The other 41 were slower or flat because the bottleneck wasn’t attention.
On batch size 1–4 with long context, arithmetic intensity stays under 40 FLOPs/byte. You’re almost always weight-bound or KV-cache bound.
The kernels that actually won did one of three things:
Persistent kernels that never re-read weights from HBM
Overlapped weight loads with previous layer compute using TMA
Aggressive KV compression (even lossy) that cut memory traffic
Spending weeks on a new attention algorithm without first hitting 65%+ of peak HBM bandwidth is usually wasted work.
What’s the biggest end-to-end speedup you’ve gotten from something that wasn’t a new attention kernel?
New open-source tool: train a model in PyTorch, then convert it into pure logic-gate hardware (Verilog/VHDL) using LUTs. Works on CPU and Apple Silicon without a GPU. Interesting step toward hardware-native neural nets.
New paper: SonicSampler — a suite of tile-aware Triton kernels that fuses the entire LLM sampling pipeline (logit processing → top-k/top-p → speculative verification) into one CUDA Graph-compatible kernel.
Key results: • Up to 10× faster top-k (hierarchical two-stage algorithm) • Up to 16× end-to-end speedup on speculative decoding vs strong baselines • Supports mixed greedy/stochastic sampling, penalties, grammar constraints, etc. in a single batched launch
Perfect for low-latency serving and EAGLE-style speculative decoding.
https://t.co/dw5yrXPvuw
Sparse MoEs make it possible to treat the SSD as part of the memory hierarchy instead of just VRAM/RAM. Magnitude’s engine keeps hot experts in GPU memory, prefetches the next ones from NVMe, and streams prefill without thrashing the cache. Result: DeepSeek V4.1 Flash at ~40+ tok/s on a single gaming GPU.
SSD is about to become the only limiting factor for local AI
Most frontier open models are ultra sparse MoEs
Qwen 3.8 Flash Next = 180B total, ~6B active per token (~3%)
GLM 5.3 Flash = 321B total, ~18B active per token (~6%)
DeepSeek V4.1 Flash = 552B total, ~16B active per token (~3%)
GLM 5.3 = 753B total, ~40B active per token (~5%)
Kimi K3.1 Pro = 2.8T total, ~104B active per token (~4%)
When ~95% of a model sits idle on every token, keeping all of it in VRAM makes no sense
Local AI has been evolving toward this for years:
2023-2025 - Layer offloading (Ollama, classic llama.cpp)
This chopped whole layers across GPU and CPU, leaving the GPU waiting on CPU once VRAM filled up
Oct 2026 - RAM expert cache (llama.cpp --moe-cache)
This shifted from whole layers to individual experts, caching hot ones in VRAM and pulling misses from RAM. Huge leap for decode, but still capped at the size of your RAM
Now - SSD expert streaming
Early tools like Colibri and Strata proved direct NVMe streaming actually works on consumer hardware. Instead of hitting a VRAM or RAM wall, your SSD becomes an active part of the engine
However most people hear "run from SSD" and assume 2 tok/s, which isn't entirely wrong. A few problems still exist:
- Basic caching kicks out experts you're about to need again
- Waiting on disk reads one layer at a time leaves the GPU and SSD sitting idle
- Getting speculative decoding to work well with SSD streaming is hard
- Long prompts are tough to keep fast without wiping out your hot cache
- Most early tools are experimental engines built for only a couple models
Making expert streaming feel fast takes solving all of those together
At Magnitude, we've been working on this for the past 2 months. Here's how we solved it:
- Smarter caching that keeps your session's most-reused experts in GPU memory
- Speculative lookahead that pulls upcoming experts off SSD before the GPU needs them
- Direct NVMe reads scheduled across layers so your SSD never sits idle
- Streaming prefill that reads each cold expert once without wiping your hot cache
- Built into a full engine that tunes itself to your chip across Macs and GPUs
All of this together means DeepSeek V4.1 Flash on a single gaming GPU at ~40+ tok/s...
We're excited to release it on Monday
GitHub here: https://t.co/5SjPQWkmrF
Helion (PyTorch’s high-level DSL → Triton) just got much better defaults.
New compile-time heuristics analyze a kernel’s IR + dataflow and pick strong configs (block sizes, warps, stages) with zero autotuning.
On H100:
1.18× vs handwritten Flash Linear Attention Triton
1.27× vs vLLM’s handwritten CUDA reduction kernels
Full autotuning only adds ~3% on top for those vLLM kernels
Autotuning used to cost hours. Now it’s tens of milliseconds.
Great for LLM inference kernels and anyone writing Triton/Helion code.
https://t.co/DpfyQuvJdI
Multi-turn thinking workflows get punished fast if every turn re-reads or re-reasons over prior context. Reserving prompt cache space upfront and reusing cached reasoning—paired with PDL kernel chaining on TensorFold—is a very practical win for local/edge multi-node setups like 2x DGX Sparks.Worth checking out the recipe here: https://t.co/Oed3OpvfPj
Big update for GLM 5.3 Flash for 2x DGX Sparks
Running latest TensorFold (v1.3.0). Lots of changes:
- Faster decode: CUDA kernels now chain directly with fewer launch gaps (PDL)
- Cleaner stack: Our paged FP8 KV, MoE, and DFlash2 speedups are now built into upstream (100+ patches down to 13)
- Smarter prompt cache: Cache space is reserved up front, and multi-turn thinking chats reuse cached reasoning instead of re-reading history
- More stable 2-GPU vision: Long image prompts no longer time out the second GPU
Even more changes and improvements coming soon!
Get it here:
https://t.co/Xein7tYXH3
Vanilla self-attention is doing too many jobs at once—local n-gram mixing, state tracking, and long-range retrieval.
Cool to see @AlexandreTL2 decouple these responsibilities in Olala mini (7B/1B MoE): Mamba-3 + Token Shift handle the local/sequential heavy lifting, leaving a sparse, 6x KV-compressed Differential Attention layer to act strictly as a high-precision recall engine.
Presenting Olala mini, a 7B total (1B active) parameter model trained on 13T tokens !
It uses a custom architecture mixing Mamba-3 with Grouped Differential Attention layers.
My work focused on architectural improvements over the Transformer, which I detail below
Nice practical speedup for S4/S4D. By rewriting the kernels in TileLang (CUDA + HIP) and adding Metal support, s4ever gets substantially faster training and inference plus lower memory use on both NVIDIA and AMD hardware, while remaining drop-in and build-free. The pure PyTorch path already matches the classic implementation, so you get the gains without losing the baseline.
Releasing s4ever: S4 and S4D SSMs with fused TileLang kernels for NVIDIA/AMD, and Metal for Macs.
Trains 4-8x faster than the original implementation and uses 2.3-2.8x less memory, with 3-5x faster inference (on an RTX A6000).
Just pip install s4ever, no build step needed.
https://t.co/S4lEa8YWZu
Interesting open-source alternative in the agent sandbox space. Cubepod runs on Kubernetes with microVM isolation (Firecracker / Cloud-Hypervisor), Overlayfs + virtio-blk for a fully persistent writable rootfs (so apt/pip installs survive), and a warm pool for fast claims. It’s designed for long-running agents, is E2B-SDK compatible, and is part of the broader CubePlex stack.
vLLM v0.34.0 lands with Model Runner V2 as the exclusive GPU runner, extensible KV cache enabled by default, and a strong Step5 NVFP4 decode improvement on GB200 systems. Another large release (1k+ commits) that continues to expand both model coverage and performance.
vLLM v0.34.0 is live! 🚀
1,061 commits · 412 contributors
🤖 10 new models
🗄️ Extensible KV cache on by default
🏃 Model Runner V2 is now the only GPU runner
⚡ Up to 1.95× faster Step5 NVFP4 decode on 4× GB200
⚠️ Upgrade notes
https://t.co/oDO3owqDZa
Interesting approach to co-residence of fine-tunes. Instead of attaching a LoRA to every matrix (which becomes prohibitively expensive when many adapters share a batch, especially with MoE), Hypertune trains a soft prefill that is prepended into the paged KV cache. This sidesteps the narrow matmul problem and is claimed to be ~75% cheaper at inference than conventional multi-LoRA serving while preserving RL quality.
Part 1: We now enter the era of hyper-composability.
For the first time, it is economically feasible to serve individually fine tuned models at scale.
And today, we open source it all.
We can now serve heterogeneous batches of custom fine-tunes, without incurring LoRA’s per matrix bandwidth overhead. This allows us to economically serve this in large batches without incurring penalties for co-residing with default model users in the same batch.
With Hypertune, we solve co-residence and are able to train co-resident adapters.
Co-residence is the ability to compose independently trained fine-tunes into one model, without capability regression from adapter interference.
Hypertune further matches LoRA in RL performance across 13 domains with 100x-500x fewer parameters.
Serving heterogenous LoRA fine-tunes in one batch is prohibitively expensive. Each fine-tune variant needs its own low-rank adapter at every matrix and is only used for a few tokens, leading to many narrow matmuls and poor utilization.
This is exacerbated for MoEs, where an adapter only engages when the token simultaneously selects a certain fine-tune and expert. This combinatorial blowup kills any remaining prospects of fast inference for fine-tuned / custom models.
Hypertune introduces soft prefill, a trained KV adapter. Soft prefill can be trained using GRPO, SFT or any other first order method.
As a consequence, Hypertune circumvents the LoRA overhead because soft prefills live entirely in the KV cache. This allows us to easily integrate it into existing serving stacks by prepending the soft prefills into paged KV blocks.
Note: Context compression using cartridges requires copious amounts of synthetically generated self-study data for context distillation. Hypertune, instead, accommodates a panoply of gradient based methods for a wider range of objectives.
Caveat: Comparisons with full-parameter GRPO are not required, because under RL, LoRA already matches it. LoRA Without Regret by @thinkymachines finds that LoRA performs equivalently to full fine-tuning for RL even at very small ranks, down to rank 1.
The takeaway:
Our gains indicate that substantial amounts of in-context learning capacity is left on the table by discrete prompts.
This could be either because soft prompts reach conditioning states that no token sequence can express, or because their continuous parameterization makes good conditioning states far easier to find via gradient descent (even if a discrete equivalent exists).
Companies and individuals can finally fine tune models on their private data, making models hyper-optimized to their needs, all without dramatically overpaying (75% cheaper than LoRA for inference).
Here's the open sourced repo: https://t.co/359woJhSEK
Part 2 coming out later this week.
A useful practical guide on trainer-generator mismatch in LLM RL. The gap between the model generating trajectories and the one being updated is easy to ignore until it starts degrading sample quality or causing reward collapse. The post walks through industrial practices for understanding, detecting, and monitoring this issue from both infra and training-loop perspectives.
A new blog, and my best industrial practice to understand, detect, monitor and debug the trainer-generator mismatch in modern LLM RL, from both infra and practitioner's perspective. Took me almost 2 weeks to write this and hope you will enjoy it.
https://t.co/kk9SiZNbcU
Decode attention just got much closer to peak bandwidth on Blackwell.
NVIDIA’s specialized decode kernel reaches ~94% of peak HBM bandwidth on GB200 (7.5 TB/s vs ~4.8 TB/s for a widely used open-source kernel).
The gains come from three changes:
Kernel designed specifically for the decode shape (q_seqlen = 1)
Better pipeline that overlaps Tensor Cores with CUDA Cores / SFU
Right choice of PTX instructions for softmax and reductions
A general attention kernel leaves a lot of bandwidth on the table during decode.
Specializing for the decode shape is still one of the highest-ROI changes.
Docs:
https://t.co/ItYJ1TocLT
Nice breakdown of the latency flip: most of the original request time was spent waiting for the first token. After the prefill context-parallel + decode data-parallel changes (plus FP8 DeepGEMM), the bottleneck shifts to the decode phase itself. Strong result on the H20-3e setup.
A team at Ant International cut DeepSeek-V4.1-Flash serving latency by about two thirds on Nvidia H20-3e GPUs.
The setup splits the work. Four GPUs handle the prompt (prefill) and four write the answer (decode), passing the cache over NVLink. The baseline used the same split with TP4 attention. Each test sent 256 requests, 24 at a time, each with about 65K tokens in and 500 tokens out.
Mean end-to-end time fell from 13.63 s to 4.70 s. Output went from 847 to 2,465 tokens per second. The biggest gain came from prefill context parallelism with decode data parallelism, then FP8 DeepGEMM scoring.
Most of the gain came before the first token. Time to first token dropped from 12.04 s to 1.89 s. Time per token after that got slower, from 3.20 to 5.62 ms.
Put those together and the wait flipped. At baseline, about 88% of each request was spent waiting for the first token. After the tuning, writing the 500-token answer takes about 60% of the time. So decode, which got slower per token, is now most of the wait.
https://t.co/0XbLbtU0KI
Interesting concept: linking multiple low-cost GPU boards over Ethernet and treating host RAM as the main memory pool with a small DRAM cache on each node. Aiming for A100-class LLM inference performance at roughly $2,400 (12 boards) is an ambitious but compelling target for personal-scale inference.
Nice release. LMCache 0.5.7.post1 hardens the defaults (non-root containers + optional API keys), expands accelerator coverage, and brings native gRPC. Solid work from 34 new contributors—just remember to check UID 1000 permissions on volumes before upgrading.
✨ LMCache v0.5.7.post1 is now available, with safer defaults, more accelerators, and native gRPC.
The official Docker images now run as the non-root user lmcache (UID 1000). The MP HTTP server, the frontend HTTP API, and the coordinator gain optional API key authentication. It stays off until you configure a key with --api-key-file.
Biren SUPA and Cambricon MLU are new platforms, and the gRPC MP transport adds native and asyncio server modes. Packages are built against PyTorch 2.14.0.
This release includes work from 34 new contributors. Thank you, and welcome.
⚙ Before you upgrade: make sure UID 1000 can read and write every path LMCache uses, including mounted volumes.
https://t.co/ku0it3BjII
#LMCache #KVCache
Nice starter guide for contributing to LMCache without a GPU. Running vLLM + LMCache end-to-end on an Apple Silicon MacBook (CPU build + NO_GPU_EXT) and verifying cache reuse makes it much more accessible for new contributors.
No GPU? You can still develop LMCache.
Our June starter guide shows a new contributor how to run vLLM and LMCache end to end on an Apple Silicon MacBook: a CPU build of vLLM, LMCache installed with NO_GPU_EXT=1, and a two-request test that shows the second request reusing the cache.
It also names four areas you can work on from a laptop: frontend, L1 eviction, L2 storage, and observability.
https://t.co/12NnZeBQNo
#ThrowbackThursday #Evergreen #Knowledge
Zig + Metal inference for a 49-layer mixture-of-experts model at 125+ tok/s on an M3 Ultra stands out. Building the whole stack without the usual ML frameworks makes this one worth paying attention to.
We wrote an LLM inference engine in Zig.
It runs Flint, a 49-layer mixture-of-experts model, on an M3 Ultra at 125.8 tokens a second. Zig + Metal. No MLX, no PyTorch, no Python in the model.
Here's one take, played back in NanoHarness at the speed it ran w graff as the harness