Nourya

Stanford BS/MS | ML Research

[Nourya @ANourya]

so excited to see this out in the world!

[Michael Y. Li @michaelyli_]

Can an LM, starting from random init (!!), learn to generate all of its pretraining data?

Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.

A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.

[Rudra Satani @rudraxx13]

@michaelyli_ self-play pretraining is just rehearsing arguments in the shower until you're right. turns out it scales.

[kfir dolev @KfirDolev]

Check our new AI paper, The Persian Rug!

https://t.co/C02MY9n1nW

We extract exactly the algorithm learned by the most well known model of neural network superposition, and distilled it into a set of weights resembling a Persian rug, which matches the learned loss exactly.