Vlad

pretraining LLMs

[Vlad @_vsavinov]

@StasBekman There’re options with cheaper inter node comms with better scalability, for example: https://t.co/CfKJRG8p3Q - maybe you’ll find it useful!

[Vlad @_vsavinov]

@cloneofsimo are there extra points for knowing that there’re e4m3 and e5m2?

[Vlad @_vsavinov]

@maharshii that’s how fsdp works

[Vlad @_vsavinov]

@giffmana https://t.co/BBu25ZbjtH

[Vlad @_vsavinov]

a no-op tensor slicing can change whether the gradients are contiguous or not..

[Vlad @_vsavinov]

@SonglinYang4 > Minimax used a very simple linear attention variant
https://t.co/bWZcs5z4AM it looks like they tried gdn as well, but maybe "we did not scale the size" is the reason behind worse perf

[Pengyu Zhao @zpysky1125]

@eliebakouch GDN hybrid is fine, Mamba2 < Mamba2 + qknorm ≈ GDN. But all those models are relatively weak in reasoning-intensive benchmarks (like BBH) compared to full-attention.
Meanwhile, for all those experiments, we did not scale the size and observe the downstream preformance.

[Vlad @_vsavinov]

rl finally works well

https://t.co/q6uULhddbY

[Vlad @_vsavinov]

https://t.co/WaKE5TGdIs

[Vlad @_vsavinov]

this comment about cuda caching allocator is what we should all strive for... almost 1000 words in a .cpp file

[Vlad @_vsavinov]

@athleticKoder you say “it doesn’t matter we’d be laid off anyway”

[Vlad @_vsavinov]

@zhongwen2009 Curious if any papers explore reweighting the reward with router statistics. For example, we can compute the router entropy for each token and assign higher rewards to decisive tokens with less uniform expert distributions (or something along those lines)

[Vlad @_vsavinov]

@m_sirovatka brain dump:
batch sizes
- https://t.co/CEUhf5rjw9
- https://t.co/YJvCgUutdb
- https://t.co/Swf1neSzsw
- https://t.co/xc3e9nQNsy
lr, schedulers
- https://t.co/V2ubjeBSh5
- https://t.co/j0lY0zT9pJ
- https://t.co/o21hLxEpi0

& tech reports actually share some details (ds v3 etc)

[Vlad @_vsavinov]

@mike64_t the only codebase where pipeline parallelism is easy to write

[Vlad @_vsavinov]

@lateinteraction there’s a dense Cohere Command A that’s been released recently

[Vlad @_vsavinov]

@icanvardar tap on space with your left hand as well, will be more symmetrical

[Vlad @_vsavinov]

@m_sirovatka https://t.co/WHEkSCJytA

[Vlad @_vsavinov]

@m_sirovatka i think this is normal, but it depends on the type of CP you’re using. if you’re using something like zig zag attention, then each CP rank would have a chunk from the beginning and the end of the sequence, so input ids are not just cut into 4 equal parts but are also shuffled

[Vlad @_vsavinov]

@giffmana @jd_markovchain what i liked about tyro is the ability to do something like https://t.co/A7YrhEITdf

[Vlad @_vsavinov]

@giffmana @jd_markovchain what about tyro?