Matthias Bal

[Matthias Bal @matthiasbal]

Post: https://t.co/Yxm3rznPIU
Code: https://t.co/mkvDH2tqui

[Matthias Bal @matthiasbal]

Nonequilibrium dynamics in spin-model transformers

We construct transformer-like computation from a driven disordered vector-spin system: inputs quench and drive the spins, mean-field relaxation implements the forward pass, and learning tunes the effective dynamics.

[NUKES @atomicarchive]

«Apache» thermonuclear test, 1.85 Megatons, Eniwetok Atoll, 9 July 1956.

[Matthias Bal @matthiasbal]

A non-equilibrium statistical mechanics perspective on transformers

https://t.co/ybu9FLDnEu

We present a class of transformers based on mean-field dynamics of vector-spin models. Our framework supports asymmetric couplings and yields residual, attention, and feed-forward terms.

[Matthias Bal @matthiasbal]

@de_gennes Haven't looked into that, but sure sounds interesting to explore. Lately I've mainly been working on a dynamical TAP-style mean-field approach to include asymmetric couplings like softmax attention into the transformers-as-vector-spin-models framework: https://t.co/dujEtJ6gVO

[Matthias Bal @matthiasbal]

New post on approaching transformer modules from statistical mechanics.

https://t.co/xs9p5RWqMn

We implement a differentiable vector-spin model whose couplings act as learnable parameters and probe it with data to find a transformer-like attention response. 1/9

[Matthias Bal @matthiasbal]

@j_bertolotti If you give it some more context, things get funkier!

[Matthias Bal @matthiasbal]

@sir_deenicus @karraaman @banburismus_ You might like this post and the previous posts it links to: https://t.co/sVEAH4wikb

[Matthias Bal @matthiasbal]

A short post on transformers and statistical mechanics.

https://t.co/FU0C4FZVTD

Can we interpret transformer modules physically? Yes, their forward pass looks like computing magnetizations from free energies in differentiable spin systems. Training shapes collective behaviour.

[Baudrillard's America @BaudrillardUSA]

Reduced pace of work, decentralization, air-conditioning, soft technologies.

[ArtNouveauDeco @NouveauDeco]

Art Deco control room of the now defunct Kelenföld power plant, built between 1927-1929, Budapest, Hungary.

[Matthias Bal @matthiasbal]

@michael_nielsen If you're into spin systems and stat mech: https://t.co/ngEk9qTjuv

[Matthias Bal @matthiasbal]

A short post on transformers and statistical mechanics.

https://t.co/FU0C4FZVTD

Can we interpret transformer modules physically? Yes, their forward pass looks like computing magnetizations from free energies in differentiable spin systems. Training shapes collective behaviour.

[Soviet Visuals @sovietvisuals]

Drama theater building in Rostov-on-Don. Photo by Mikhail Prekhner, USSR, 1937

[Matthias Bal @matthiasbal]

A short post on transformers and statistical mechanics.

https://t.co/FU0C4FZVTD

Can we interpret transformer modules physically? Yes, their forward pass looks like computing magnetizations from free energies in differentiable spin systems. Training shapes collective behaviour.

[Matthias Bal @matthiasbal]

@Thom_Wolf Have you read Tomasello's Becoming Human? It's fascinating.

[Soviet Visuals @sovietvisuals]

"Kosmos" cafe, Perm, USSR, 1960s

[Matthias Bal @matthiasbal]

Our post comes shipped with an experimental PyTorch implementation: https://t.co/f8hzyZZLE7 Comments welcome! 9/9

[Matthias Bal @matthiasbal]

We previously interpreted the deep equilibrium fixed-point procedure physically as solving the mean-field fixed-point equations of a vector-spin model. This work instead takes derivatives of the steepest-descent partition function of a particular class of vector-spin models. 8/9