"Contextual Correction of Parallel Drafts for Speculative Decoding"
Parallel speculative decoding can generate multiple tokens at once, but the tokens can be inconsistent because they don't know which earlier tokens were selected.
This paper DCOCO uses a lightweight corrector to score token combinations in parallel, improving draft accuracy without the expensive sequential computation.
Across four LLMs, it achieves up to 6.6x faster decoding than standard generation and 8-19% higher throughput than competing speculative decoding methods.
https://t.co/VkPBifhWDm
A really clever architecture for diffusion-based speculative decoding!
Huge kudos to @roy_eisenstadt and @IlanNaiman for driving this from start to finish with remarkable initiative and ownership - Impressive work!
Proud to be part of this team! ๐
1/7 ๐ฅฅ New speculative decoding method!
Introducing DCOCO: Contextual Correction of Parallel Drafts.
Parallel drafters are fast, but lose context between predicted tokens.
DCOCO restores it without giving up parallelism.
Up to 6.6ร faster than vanilla decoding. ๐งต