Zachary Charles

distributed machine learning @ google | sometimes mathematician

[Jeff Dean @JeffDean]

It's been a delight to provide small amounts of advice and suggestions to people working on the Decoupled DiLoCo training system. This approach enables graceful handling of failures in large scale training jobs, by allowing (N-1) / N units to proceed when one fails.

Thread ⬇️

[Arthur Douillard @Ar_Douillard]

The DiLoCo team at Google DeepMind and Google Research is proud to release Decoupled DiLoCo, the next frontier for resilient AI pre-training.

Decoupled DiLoCo enables training with datacenters across the world, using heterogeneous hardware, and never halting the system despite hardware failures.

[Zachary Charles @MatharyCharles]

How do people write math papers with AI these days? What scaffolding/agents/editors do people find helpful?

[Zachary Charles @MatharyCharles]

To clarify: I think the paper is cool! But the idea of applying JL transformer and random rotations for compression is a very well understood mechanism

[Zachary Charles @MatharyCharles]

Compression for gradients (or models in distributed training) have been using these techniques for decades.

I'm sure there are nuances to what properties you want for KV cache compression but all of these techniques are basically well trodden.

[Arya Mazumdar @MountainOfMoon]

@yoavgo @BlackHC Sorry, this entire discourse is a bit laughable, because 100s of papers uses random projection followed by sign quantization. There is no need to call it q JL, because there are important differences and this goes by the name of 1-bit compressed sensing, hyperplane hashing etc

[Arya Mazumdar @MountainOfMoon]

@yoavgo @BlackHC Sorry, this entire discourse is a bit laughable, because 100s of papers uses random projection followed by sign quantization. There is no need to call it q JL, because there are important differences and this goes by the name of 1-bit compressed sensing, hyperplane hashing etc

[Zachary Charles @MatharyCharles]

This exodus has been brutal, so many people leaving whose work I have thoroughly appreciated over the years

[Luca Soldaini 🎀 @soldni]

After 4yrs, today is my last day at @allen_ai

It was an honor to work on Olmo, Dolma, olmOCR, Tulu, Molmo & other fully-open artifacts 🫡 Reception has been amazing & their adoption makes me SO PROUD 🥹

Team is super committed to open recipes; can't wait to see what's next!!!!

[Zachary Charles @MatharyCharles]

@dlwh Oh very cool to see hyperball in action! Did you also normalize the updates?

[Zachary Charles @MatharyCharles]

@Dorialexander Tbh I might even bet that one could make RNNs + SOTA data better than SOTA transformers + old data.

[Zachary Charles @MatharyCharles]

@Anthony_Bonato To be fair this is a very important step in the mathematical process

[Zachary Charles @MatharyCharles]

Fun fact: you can use this to easily show that for any finite field F and degree n, there is an irreducible polynomial of degree n with coefficients in F.

[Algebra Etc. @AlgebraFact]

For every field F, there is a an algebraically closed field K that contains F.

[Zachary Charles @MatharyCharles]

@kalomaze Baba is AGI is honestly not the worst benchmark I can think of

[Zachary Charles @MatharyCharles]

This is just not true. Crack open a copy of Roger & Horn and behold the many, many non-obvious facts about linear algebra.

[little grey mouse 🐭 @mouse_math]

probably more than any other field of mathematics, linear algebra strikes me as a bunch of trivial (ie obvious) facts, yet the end result is impressively powerful.

[Zachary Charles @MatharyCharles]

This is a very good paper, highly recommend.

My one gripe - the single learner DiLoCo case (called SNOO in the meta paper) has been known to improve AdamW well-before it was called SNOO, e.g. see:
* https://t.co/DMWYyc6cQA - Table 3
* https://t.co/9iCIk8oZEb - Figure 2 & 3 (and more)

I guess we should've given it a fun name to take credit for this?

[Hao-Jun Michael Shi @hjmshi]

1/10 Are DiLoCo and Schedule-Free actually related? A brief history and unusually late advertisement for our work: Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs (see https://t.co/ESaSU8kwpx).

[Zachary Charles @MatharyCharles]

@anpaure @Romy_Holland I think it unfortunately gets worse the more specialized a topic is. Wikipedia is famously bad for many math concepts, and in other fields is often used to drum up attention to people's (bad) papers.

[Zachary Charles @MatharyCharles]

@jxbz @yuxiangw_cs Very cool! Can you point me to anything that talks about what you referenced regarding signSGD and compute optimality for LLMs?

[Zachary Charles @MatharyCharles]

@eliebakouch my job as an AI researcher is preventing me from spending time learning about AI research

many such cases

[Zachary Charles @MatharyCharles]

@natalienkhalil @arxiv Why is it imperative to have survey papers allowed on arXiv? There are so many ways to disseminate such things, whereas in many fields of math (for example), new submissions to arXiv are used as a signal for what to read in a given day.