Parth Shroff

Stanford (MS '26 - ECE) - [email protected]
UT Austin (BS '24 - CS (Turing), ECE, Math)

[Parth Shroff @pshroff_]

Context-Sharded Block Parallelism (CSBP) speeds up long-context Diffusion LLMs via clean context sharding & block load balancing—up to 7.59× speedups! 🚀

Open-sourced in Turbo-dLLM.

W/ @TarunSures41845 @hangoo_kang @PranshuChatur11 @ishanskhare @KumbongHermann @Azaliamirh

[Tarun Suresh @ COLM 2026 @TarunSures41845]

Diffusion LLMs and speculative decoding promise much faster agents. Yet agents need long contexts, and training on them is painfully slow.

Introducing Context-Sharded Block Parallelism (CSBP), a new distributed parallelism strategy unlocking significant training efficiency for diffusion LLMs, with speedup gains growing with context length 🚀

⚡ 7.59× faster DFlash2 speculative decoding drafter training
⚡ 1.61× faster block diffusion fine-tuning
⚡ 1.33× faster autoregressive → block diffusion adaptation

With the same GPU hours, models trained with CSBP score higher on SWE-bench Verified and Terminal-Bench Lite 👑

Open-sourced in Turbo-dLLM, our new optimized distributed training library.

Advised by @Azaliamirh and with an amazing team: @PranshuChatur11 @hangoo_kang @pshroff_ @ishanskhare @KumbongHermann