Speculative Decoding Clearly Explained
(done right, it can make LLM inference 4x faster)
A local LLM typically produces one token per model pass. Each pass reads the model’s weights from memory, often leaving compute units waiting for data.
Generating a sentence means repeating that expensive process for every token.
Now introduce a smaller model that can predict the next token quickly. Let it make several predictions ahead while the larger model waits to check them.
The smaller model is the draft model. The larger model, called the target model, is the LLM whose output we want to preserve.
The draft model proposes. The target model verifies.
Because the guesses are already available, the target can check several positions in one pass instead of generating each token separately.
Here’s how that works.
→ Draft a possible continuation.
After “The cat sat,” the draft model proposes “on the red mat.” These tokens are provisional.
→ Check the guesses together.
The target model scores all four positions in one forward pass. Each position uses the preceding draft tokens as context.
→ Accept from left to right.
For sampling, let p be the target model’s probability for a guess and q the draft model’s probability. Accept it with probability min(1, p/q), stopping at the first rejection.
→ Correct and continue.
Discard the rejected token and all later guesses. Sample a replacement from the positive difference between the target and draft distributions, normalized into probabilities.
In the diagram, “on” and “the” are accepted. “Red” is rejected, and the replacement is “big.”
That produces three new tokens from one target-model pass. If every guess is accepted, the target can supply an additional token.
This acceptance and correction procedure preserves the target model’s output distribution. The speedup comes from replacing expensive passes with cheaper drafting and shared verification.
Uzu, an open-source engine for Apple Silicon, takes this further. DFlash proposes candidates in parallel, Weaver builds coherent draft paths, and rollback-free verification checks branches without repeatedly restoring the model’s running state.
In our base M5 coding test, 4-bit Qwen3.5 9B reached 92 to 117 tokens per second with Uzu, compared with 22 using plain llama.cpp. These complete engine results also reflect quantization and hardware-specific kernels.
Explore Uzu on GitHub: https://t.co/CayueSO3B9
(don't forget to star 🌟)
My quoted article explains how Uzu combines these techniques and how to benchmark the result on your own prompts.
fun interview. we cover:
- story of muse
- why my 19th bday attendees are now all billionaires
- diamond mining vs building skyscrapers
- my 200+ direct reports
VISUALIZE MASSIVE GIT REPOSITORIES WITH TOUCH! 🌲
Markrai GitpowVisualize large git repositories with GitPow using image previews and touch navigation.
https://t.co/4UHETdael2
SELF HOSTED OPENROUTER WITH SMART FALLBACKS! 🔀
Continuum Ai Corp Orcarouter LiteSelf-hosted OpenRouter alternative with managed fallbacks
https://t.co/09xL11na8Y
EXTRACT TEXT FROM IMAGES ON MACOS! 🔍
Samirpatil2000 BufferExtracts searchable text from images using on-device Vision and manages local clipboard history on macOS.
https://t.co/e2XGI5GrqA
DESTROY DUPLICATE FILES ACROSS YOUR DRIVES! 🧹
Pkolaczk FclonesRemoves duplicate files across directories while preserving one copy per group.
https://t.co/xXyFaYWoND
RUN ULTRA FAST LLMS WITHOUT PYTORCH! ⚡
Zyora Dev ZseRuns LLM inference without PyTorch, Triton, or transformers, compiling kernels directly to GPU code for rapid model loading.
https://t.co/PdaYxp8GsT
GENERATE ACCURATE DEPTH MAPS FROM IMAGES! 👁️
Aigeeksgroup AnydepthAnyDepth generates depth maps from single images using the Simple Depth Transformer decoder.
https://t.co/dklDyXr8gy