John Rose

Hex enjoyer. Linux/C/CUDA

[John Rose @jrose2000]

@MainzOnX Fix H, D per model. Bucket S to powers of 2. Let XLA pad each bucket. Cache compiled HLOs keyed by bucket. Compile on cold start only. Amortized over 000s of requests, the recompile cost vanishes

[John Rose @jrose2000]

@MoritzW42 Mythos pls fix think hard no mistakes surprise me

[John Rose @jrose2000]

@roydanroy Don’t tempt me with a good time

[John Rose @jrose2000]

From shakespeare-char up to enwik8 for the depth testing

[John Rose @jrose2000]

Layer-wise training usually collapses with depth. DiffusionBlocks x NanoGPT breaks the mold. Gap to baseline flat from L=6 to L=12, VRAM ~half

[John Rose @jrose2000]

@ritv3999 Ah thanks, though I wanna pressure test further. How does the σ-shuffle hold up to more depth!?

[John Rose @jrose2000]

@mattmireles Sorry for your loss

[John Rose @jrose2000]

@ritv3999 Nice! I landed in the 0.45 hood as well. Definitely been interpreting "works at all" as exciting. Curious what your arch choice was for where the noised target enters.

[John Rose @jrose2000]

@ritv3999 Exciting: possible to train a causal AR language model with independent blocks and no gradient flows

[John Rose @jrose2000]

@ritv3999 Early numbers: 6-layer causal GPT, 3 independent blocks. ~2.16× lower peak VRAM, val CE within ~0.6 nat of baseline. Costs more training steps to converge

[John Rose @jrose2000]

@CathPoaster Relatable. Have you considered naming your stack

[John Rose @jrose2000]

https://t.co/62iEgJMDvt

[John Rose @jrose2000]

@ruima Disclaimer: wild speculation

[John Rose @jrose2000]

@ruima If all TPUs, v5e assume 5000 tok/s, total ~12k TPUs at 100% util, or ~24k TPUs reqd with traffic/redun