Mahbod Majid

[Jacob Springer @jacspringer]

How would you design a pretrained LLM that preserves output diversity AUTOMATICALLY after finetuning?

Our method: learn a diverse “annotation” distribution from the pretraining data that conditions the generations, and then **don’t touch it when fine-tuning**!

1/

[Gavin Brown @gavinrbrown1]

You have: dataset X, query access to some function f

You want: a differentially private estimate of f(X)

A thread on new work with Ephraim Linder, Mahbod Majid, and Vikrant Singhal. (1/9)

[Jacob Springer @jacspringer]

Just released a new pretraining paper with some interesting takeaways:
- sharpness minimization is important but it doesn’t show its benefit until *after* you post-train
- increase your learning rate!! (this is free!)

read Ishaan’s thread but I’ll also add my 2 cents below
1/n

[Konwoo Kim @konwookim]

for data-constrained pre-training, synth data isn’t just benchmaxxing, it lowers loss on the real data distribution as we generate more tokens

for even better scaling, treat synth gens as forming one long 𝗺𝗲𝗴𝗮𝗱𝗼𝗰: 1.8x data efficiency with larger gains under more compute

[Suhas Kotha @kothasuhas]

Since compute grows faster than the web, we think the future of pre-training lies in the algorithms that will best leverage ♾ compute

We find simple recipes that improve the asymptote of compute scaling laws to be 5x data efficient, offering better perf w/ sufficient compute

[Gautam Kamath @thegautamkamath]

I'm in Boston for one day for a privacy workshop (TPDP 2023). @zakynthinou and @mahbodm_ present the first contributed talk, on showing how robust estimators imply private estimators. Check out both works! (https://t.co/T7UhkPlX2Y, https://t.co/k8fsj4SVJq)

[Mahbod Majid @mahbodm_]

Just received the Faculty of Math Graduate Research Excellence Award from @WaterlooMath and I couldn't be more grateful! Huge thanks to my amazing advisor Gautam for the guidance and support.

[Gautam Kamath @thegautamkamath]

Congrats to Mahbod (@mahbodm_)! If you want to read more about the work, check out this thread (https://t.co/xbSbHD9cVo).

[TCS blog aggregator @cstheory]

Robustness Implies Privacy in Statistical Estimation https://t.co/5tXul2hbp0

[Mahbod Majid @mahbodm_]

This is in some sense my debut album so I’m feeling pretty excited about it.

[Mahbod Majid @mahbodm_]

Check out my talk about our paper "Efficient Mean Estimation with Pure Differential Privacy via a Sum-of-Squares Exponential Mechanism" (https://t.co/0FFgSTv79p) for STOC 2022.

Based on join work with @Samuel_BKH and @thegautamkamath.

https://t.co/N1R82PX9TI

[Gautam Kamath @thegautamkamath]

New paper on arXiv: "Efficient Mean Estimation with Pure Differential Privacy via a Sum-of-Squares Exponential Mechanism," with @Samuel_BKH and @mahbodm_.

Finally resolves an open problem on my mind since April 2019 (~2.5 years ago).

https://t.co/K8xWukFO6O
🧵Thread ⬇️ 1/n