Wei (Will) Feng

PyTorch Distributed, FSDP, float8

[Wei (Will) Feng @weifengpy]

@MaziyarPanahi @axolotl_ai great work!

[Wei (Will) Feng @weifengpy]

@iamgrigorev make a request at torchao nvfp4 and we can talk about this https://t.co/DE8fMZNnZF

[Wei (Will) Feng @weifengpy]

@MaziyarPanahi verl uses torch==2.6.0 for fsdp2 but need some patch on state dict to address oom https://t.co/dnrifmtpu3

[Wei (Will) Feng @weifengpy]

@Infopulsed @maharshii the torchtitan way is apply AC, apply torch.compile, apply fsdp2. let fsdp2 hooks execute in eager and let forward execute in compile. but if you really need full graph capture, simplefsdp is our best offer https://t.co/ByHWwfPPxI

[Wei (Will) Feng @weifengpy]

@EdmondD52582093 @HaHoang411 @_lewtun that's good to know. verl and nemo-rl both support fsdp2. might be helpful to you

[Wei (Will) Feng @weifengpy]

@EdmondD52582093 @HaHoang411 @_lewtun it's my pleasure. feedbacks are always welcomed

[Wei (Will) Feng @weifengpy]

@EthanBThoma @madhav1 which bug are you running into for fsdp2?

[Wei (Will) Feng @weifengpy]

@m_sirovatka @TheZachMueller sounds good. thanks for giving it a try on fsdp2

[Wei (Will) Feng @weifengpy]

@m_sirovatka @TheZachMueller could I know the pain point?

[Wei (Will) Feng @weifengpy]

@RameshArv1nd @intervitens the speedup is real when it’s compute bound

[Wei (Will) Feng @weifengpy]

@danielhanchen @liloba994127 @UnslothAI Hi Daniel, let me know if you got any blockers around fsdp2. we can facilitate the enablement

[Wei (Will) Feng @weifengpy]

@bozavlado Hey Vlado, do you find fsdp2 code hard to understand?

[Wei (Will) Feng @weifengpy]

@errantdata could I know more about 2) fsdp2 + autograd. is it resize_(0) that bothers you?

[Wei (Will) Feng @weifengpy]

@Infopulsed @wanchao_ @giffmana @__kolesnikov__ @XiaohuaZhai we relaxed the assertion for int dtypes in pytorch nightly. if int dtypes paramters are not trainable, they can be mixed in a single all-gather

[Wei (Will) Feng @weifengpy]

@YouJiacheng @qsh_zh @JingyuanLiu123 @SeunghyunSEO7 @main_horse we are thinking about this. one possibility is customized reduce-scatter. perform GGT computation on unsharded gradients, and scatter them afterwards. probably will land some hooks into pytorch and examples to titan

[Wei (Will) Feng @weifengpy]

@m_sirovatka Hi Matej, fsdp2 maintainer here. do you have script for repro?

[Wei (Will) Feng @weifengpy]

@tokenpilled65B @andersonbcdefg @marksaroufim Hi Philippe, we just published FSDP2 tutorial https://t.co/yiNOW58hBP
welcome your feedback

[Wei (Will) Feng @weifengpy]

@andersonbcdefg @marksaroufim Hi Ben, we published FSDP2 tutorial https://t.co/yiNOW58hBP
Thanks for reporting the problem to us