Garrett Goon

AI at IBM Research.

[Garrett Goon @GoonGarrett]

@FilipoGiovanni @willccbb Had the same reaction

[Garrett Goon @GoonGarrett]

@willccbb @eliebakouch Very good. Claude's writing was entirely fine, would be happy to see more

[Garrett Goon @GoonGarrett]

@StefanGliga @ezyang Why do you want it to die? What's your preferred alternative?

[Garrett Goon @GoonGarrett]

@PatrickToulme @ezyang Also my expectation

[Garrett Goon @GoonGarrett]

@WentaoGuo7 Awesome, congrats! Have you gotten to test if the fully_shard hangs still occur with the update? I could try tomorrow if not.

[Garrett Goon @GoonGarrett]

@wightmanr @jeremyphoward Great call out.

I was also searching recently for what init schemes are being used in recent work and couldn't find much. Arcee's Trinity tech report was one find. Can you suggest others in addition to Olmo?

[Garrett Goon @GoonGarrett]

@m_sirovatka I hit a bunch of "CUDA unspecified launch errors" recently. New one for me. Very unwelcome

[Garrett Goon @GoonGarrett]

@stochasticchasm I guess there's a middle ground where you compute the loss itself in chunks but realize the full logits as usual. Some memory savings + logits/KL unaffected

[Garrett Goon @GoonGarrett]

@stochasticchasm Ah ok, got it. Surprised about the instabilities mentioned elsewhere. What ctx len do they start to be significant at?

[Garrett Goon @GoonGarrett]

@difficultyang So many headaches with CC rendering. Hitting this one often. Also seems exacerbated by tmux

[Garrett Goon @GoonGarrett]

@YouJiacheng What's the strategy to counteract this, then?

[Garrett Goon @GoonGarrett]

@m_sirovatka @samsja19 <2000 tokens? 🤔

[Garrett Goon @GoonGarrett]

@eliebakouch Wow, the Sonnet jump @256k

[Garrett Goon @GoonGarrett]

@StasBekman @aryanvs_ The speed up was due to throwing more GPUs at the workload, though, right? Baseline being a single GPU, other cases using N, and seeing a near linear decrease in overall runtime? Not a per-GPU improvement

[Garrett Goon @GoonGarrett]

@m_sirovatka @oneill_c @Jozef_Nathaniel Nightly or GTFO (saw the other threads before this one)

[Garrett Goon @GoonGarrett]

@m_sirovatka Yeah agreed, just thinking about how you could solve it if you really needed to

[Garrett Goon @GoonGarrett]

@m_sirovatka Add like a fully_shard_bwd API to wrap modules whose grads should be RS'd together. E.g. fully_shard on a whole transformer block and wrap the MLP and attn sub-blocks each with fully_shard_bwd so their grads are bucket-reduced and freed earlier than default. Maybe too complicated

[Garrett Goon @GoonGarrett]

@m_sirovatka Yeah was thinking the same. Basically enabling the backwards bucketing strategy to be different from the fwd, rather than forcing both to consolidate all collectives into a single launch

[Garrett Goon @GoonGarrett]

@stochasticchasm @m_sirovatka Same, yeah. Still annoying though.