Davis Blalock

This account is just so I can reply to people without spamming all my followers' feeds as @davisblalock. Also for twitter API testing.

[Davis Blalock @dblalock_debug]

@SemiAnalysis_ rare photo of jensen next to someone more legendary than him

[Davis Blalock @dblalock_debug]

@GaryMarcus Are there actual people saying "scaling is all you need"? (Excluding randos outside the field, who of course have all sorts of crazy takes)

[Davis Blalock @dblalock_debug]

@Clashluke ...I see you have some tests sanity checking the ECC, but we also got lots of negative results like this until we really tested and debugged the crap out of it.

I'd say to worry if the steps sizes are within ~2^-14 of the param values and it doesn't look like baseline f32. [2/2]

[Davis Blalock @dblalock_debug]

@Clashluke Fun fact: we actually wanted to add stochastic rounding to FlashOptim, but couldn't get ablations to justify it since the 24-bit version was iso quality in all our evals.

If you have a simple repro, that would be awesome.

[Davis Blalock @dblalock_debug]

@Clashluke Relatedly, the ECC code was incredibly hard to get right and ended up depending on a ton of details we couldn't trust torch (or even triton) to implement properly. This is most of why we ended up just brute force measuring all the bitstrings.. [1/2]

[Davis Blalock @dblalock_debug]

@bozavlado wait, all of them? (Did we miss related work?) Or is just that there are other techniques with similar vibes?

[Davis Blalock @dblalock_debug]

@BlackHC @davisblalock Yes, similar to the CDF. The CDF itself isn't optimal though. You ideally want bin boundaries spaced proportional to a power of the density, depending on your distortion metric.

And in practice you just pick some function f() where f() and f^1() run in a few cycles.

[Davis Blalock @dblalock_debug]

@benjamin_warner The optimi guy! I think you're cool and your library is cool.

[Davis Blalock @dblalock_debug]

@YouJiacheng Fair. Although 1) I've mostly seen people do Gaussian bins via table lookups/NF4, and 2) I want to call attention to the fact that there's a whole design space here, not just a trick for Gaussians. I don't think most people realize much/most of the quant alpha is in preprocessing

[Davis Blalock @dblalock_debug]

@YouJiacheng Yeah, you might not want bf16 accumulators for your gradients. The good news is that this is an allreduce choice that can be decoupled from the format you keep the final grads in (although IIRC the released code couples these...).

[Davis Blalock @dblalock_debug]

@gaunernst @YouJiacheng Fun fact: the initial impetus for the weight splitting was just to work around this limitation.

[Davis Blalock @dblalock_debug]

@GoonGarrett @davisblalock thanks! will hopefully get back to it someday...

[Davis Blalock @dblalock_debug]

@YouJiacheng Accurate. IMO you can never be sure that any quantization will preserve quality, including ours, until you've tried it on your own scaling ladder in your own codebase.

[Davis Blalock @dblalock_debug]

@prajdabre @davisblalock @Tim_Dettmers we also aren't really claiming to take the crown of fewest possible bits of optimizer state. The claim is more "look, we do better end-to-end by shrinking lots of different tensors" plus some "look, you can get away with surprisingly simple stuff for the optimizer state part"

[Davis Blalock @dblalock_debug]

@prajdabre @davisblalock @Tim_Dettmers negative. This is one of those results that we wanted to have, but didn't prioritize because it wasn't clear what it would demonstrate. If we're already matching the ground truth as well as we can measure, unclear what comparison to another approximation of the ground truth shows

[Davis Blalock @dblalock_debug]

@soldni Accurate take: both Jose and I were EE undergrad.

[Davis Blalock @dblalock_debug]

@itsclivetime @davisblalock So I'm not sure, but would be really cool if this could help simplify the current logic (https://t.co/o8n7kKpckf).

[Davis Blalock @dblalock_debug]

@itsclivetime @davisblalock ...ones you pointed out, there's also a bunch of weirdness with subnormals and cases where the downcast value is subnormal while the f32 is still normal.

Getting all this right took a lot of thought and debugging, so I'm not sure a priori how the ^ (1 << 15) would play with this

[Davis Blalock @dblalock_debug]

@itsclivetime @davisblalock Alright I thought about this a little bit in the case of 4 bits being dropped and I think (?) this basically works for encoding where within [val_min, val_max] you are as an unsigned int.

What's less clear are all the edge cases. Besides the...

[Davis Blalock @dblalock_debug]

@khoomeik This exists. Deep learning people just haven't read it. And the recent stuff isn't published.