VE:D

ML infra @axolotl_ai

[VE:D @ant_vedaya]

here you go spec decoding under 3k lines ,

[VE:D @ant_vedaya]

writing a speculative decoding engine from scratch in ~3k lines of PyTorch + FlashInfer. Paged KV, continuous batching, CUDA graphs, EAGLE-3 chain + tree. 1.94× at batch 1 on an H100, every output byte-identical to greedy. Blog coming soon

[VE:D @ant_vedaya]

@HimanshuSh58438 @tenstorrent < 3

[VE:D @ant_vedaya]

Axolotl v0.21.0 is out!

501B MoE fine-tuning on 8xH200. Native diffusion LM training. Multimodal DPO, KTO & GRPO. 8% faster eager LoRA. CoreWeave cloud training.

Missed v0.20.0? GGUF exports, NVFP4 LoRA, long-context training & more.

Which feature are you trying first?

Release notes: https://t.co/d8HWGV6PP9

[VE:D @ant_vedaya]

@pikoreels Crazyyy

[Wing Lian (caseus) @winglian]

Using BFD, you end up with longer samples getting paired up with smaller samples. Depending on the correlation of tasks to sample lengths, this can lead to some pairing off of tasks within a batch (at smaller effective batch sizes).

[VE:D @ant_vedaya]

Try it yourself: https://t.co/qh92u1obYo

Full method, every run and raw numbers: https://t.co/RcsL6FU66z

[VE:D @ant_vedaya]

coming up with a big release soon !

So benchmarked fine-tuning GLM-4.7-Flash on a single H100 against three other popular training frameworks.

Same model, same data, same 21M LoRA params, 32k tokens per step.

[VE:D @ant_vedaya]

Long context: 32k-token sequences on a single 80 GB H100 with offloading, and 64k with one more line:

activation_offloading: hidden_states

[VE:D @ant_vedaya]

@HimanshuSh58438 cooool

[Wing Lian (caseus) @winglian]

If you're curious why they only reported against TRL for their single GPU SFT on GLM Flash, it's because at that regime, they're still slower than Axolotl by about 5% (can't find any B300s, so using B200s). But hey, reporting benchmarks these days is just about cherry-picking data, right?

[VE:D @ant_vedaya]

on to some personal research.

speculative decoding inside GRPO rollouts.
the Question: how lossy can you make draft acceptance before RL training breaks?

answer: very lossy, if the trainer knows what the sampler did.

x axis is rollout throughput, y is reward after 200 steps. blue = lossy acceptance, no fix. red = same acceptance + the fix. the long arrow is the whole paper: 26% faster than lossless, same reward.

preprint sooon. rollout engine is nanospec, GRPO loop is ~400 lines, all on one H100 for under $50.

[VE:D @ant_vedaya]

https://t.co/cw5aMPsIg9

[VE:D @ant_vedaya]

writing a speculative decoding engine from scratch in ~3k lines of PyTorch + FlashInfer. Paged KV, continuous batching, CUDA graphs, EAGLE-3 chain + tree. 1.94× at batch 1 on an H100, every output byte-identical to greedy. Blog coming soon

[VE:D @ant_vedaya]

code: https://t.co/gBu2l463HS

[VE:D @ant_vedaya]

https://t.co/cw5aMPsIg9

[VE:D @ant_vedaya]

here you go spec decoding under 3k lines ,

[VE:D @ant_vedaya]

writing a speculative decoding engine from scratch in ~3k lines of PyTorch + FlashInfer. Paged KV, continuous batching, CUDA graphs, EAGLE-3 chain + tree. 1.94× at batch 1 on an H100, every output byte-identical to greedy. Blog coming soon

[VE:D @ant_vedaya]

@HimanshuSh58438 🫡

[VE:D @ant_vedaya]

https://t.co/b0fZzopYyR

[VE:D @ant_vedaya]

Is there another trainer that composes EP × CP × FSDP2 on unmodified HF transformers models? prime-rl and torchtitan do it with their own modeling code. Axolotl does it by patching the experts path in transformers. Curious what else exists.