M. G. Mashiku

[Sakana AI @SakanaAILabs]

Introducing PC-ALM, a local-learning alternative to backpropagation.

Our method trains 1000-layer neural nets using only local dynamics, and without backprop.

Blog: https://t.co/bBGCgalqKW

Standard deep learning relies on backpropagation. The brain, however, cannot implement backpropagation, at least not exactly. How can a physical system, such as the brain, solve multilayer credit assignment without explicit use of backprop?

We look for inspiration in two related fields: distributed optimization and NeuroAI.

In NeuroAI, predictive coding asks each neuron activation to solve an energy-based inference problem instead of using a standard forward pass. That inference step can be implemented as energy-minimization dynamics on local prediction errors.

This perspective -- each layer as a dynamical system -- has proven promising, but performance of predictive coding hasn't scaled well with depth. Credit signals at far ends of the network struggle to diffuse into internal layers.

We turn to distributed optimization, generalizing predictive coding to use an augmented Lagrangian instead of energy. This motivation stems back to a classic 1988 paper by LeCun, showing that the Lagrange multipliers of a deep network can be identified with gradients of a supervised loss. The augmented Lagrangian then bridges LeCun's perspective to the standard predictive coding that is used in NeuroAI.

We find that this new perspective yields a natural PC-like alternative to backpropagation, resulting in a method we call PC-ALM. PC-ALM differs from PC in that it introduces dual neurons (Lagrange multipliers) as part of the layer-local dynamics, resulting in each layer acting as a PI feedback control system to minimize local prediction errors.

We find that PC-ALM is capable of propagating signals to seemingly arbitrary depth, especially in deep narrow networks where standard PC struggles to learn.

Ultimately, our motivation here is to understand how distributed physical systems, such as the brain, can compute credit signals using only local coupling and local dynamics.

PC-ALM may also inform deep learning in neuromorphic hardware, where dynamics are cheaper than on GPUs.

Paper: https://t.co/doSZ8mzoyK
Code: https://t.co/rxEDIszVKD

[Tai-Danae Bradley @math3ma]

I’m excited for this new article in the Notices of the AMS (@amermathsoc). In it, we describe several nice analogies between linear algebra and category theory and discuss connections to language and LLMs. Coauthored with John Terilla and @gastaldi_gianni.
https://t.co/rd3BB0MKOa

[Hannes Stark @HannesStaerk]

Tomorrow @HoldijkLars presents his paper "Path Integral Stochastic Optimal Control for Sampling Transition Paths" (https://t.co/vV5gndJP3R)

I think the SOC ideas might have even more applicability in this field!

Join on Zoom at 11am EDT / 5pm CET: https://t.co/kDchkDIf4U

[Prof. Brian Keating @Briankeating]

Sir Roger Penrose | The Emperor’s New Mind: Consciousness & Computers | @Into_Impossible
📺: https://t.co/PxblQqysdg

[Yann LeCun @ylecun]

<not_an_april_fool_joke>
By amplifying human intelligence, AI may cause a new Renaissance, perhaps a new phase of the Enlightenment.
But prophecies of AI doom are also causing a new form of medieval obscurantism.
</not_an_april_fool_joke>

[Jonathan Gorard @getjonwithit]

https://t.co/QSwyHdIfGi
ICYMI, I've spent the last several months developing "Gravitas": a new computational relativity tool based on @wolframphysics formalism, enabling anyone to perform hypergraph-based numerical relativity simulations on their laptop. (1/4)

[Antonio Ortega @elpenta]

I'm very happy to share that my book has now been printed and will be available next week!

Introduction to Graph Signal Processing: https://t.co/TyofgAb7Il

[Steven Brunton @eigensteve]

Numerical Integration to Simulate Nonlinear Dynamics! The final phase of our series on Differential Equations

https://t.co/ahLVeKAeYj

Topics
#Chaos
Forward/backward Euler
Stability & accuracy
Runge-Kutta
Symplectic & variational integrators
Uncertainty propagation
Code examples

[Petar Veličković @PetarV_93]

Excited to share our generalist neural algorithmic learner! 📚🤖

One graph neural network🕸️, one set of parameters🌐, *thirty* diverse algorithmic tasks 💻 -- delivering on our promise from Oct'21.

It's been quite an amazing journey 🚀
Personal reflections in the thread... 🧵

[M. G. Mashiku @mashiku_m]

@DStrachman The Paper Kites

The Oh Hellos

Night Travellers

[Max Welling @wellingmax]

It’s time to add a 6th G to the 5 G's of geometric DL: Grids, Groups, Graphs, Geodesics, Gauges and Geometric Algebra. (@TacoCohen, @mmbronstein
@joanbruna, @PetarV_93)
Congratulations to a fantastic team: @jo_brandstetter, @rejuvyesh, @vdbergrianne.

[M. G. Mashiku @mashiku_m]

Google AI Blog:Minerva: Solving Quantitative Reasoning Problems with Language Models https://t.co/QPEkz3I8yR via @googleai

[Petar Veličković @PetarV_93]

In our latest work (to appear at two #ICLR2022 workshops!), Andrew Dudzik & I use category theory and abstract algebra to more precisely characterise how GNNs align to dynamic programming.

This further generalises the great work of @KeyuluXu et al!

https://t.co/IOEMlYStJg

[M. G. Mashiku @mashiku_m]

Thankful to have found such an insightful conversation between @ilangur and @Ben_Reinhardt.
Shaping Research by Changing Context with Ilan Gur [Idea Machines #36] https://t.co/zNXmI0u7l1

[Kevin Patrick Murphy @sirbayes]

I am delighted to announce that a draft of my latest book, “Probabilistic Machine Learning: Advanced Topics”, is now available online at https://t.co/dSlKkwYpLr. It covers #DeepGenerativeModels, #BayesianInference, #Causality, #ReinforcementLearning, #DistributionShift, etc.

[Jonathan Gorard @getjonwithit]

https://t.co/sFTJ8hykMI
New paper (with Xerxes) on the homotopic foundations of @wolframphysics , addressing the fundamental question of *how* and *why* geometrical structures in physics (such as spacetime, Hilbert space, etc.) emerge from discrete "pregeometric" data! (1/5)

[Stanford AI Lab @StanfordAILab]

Congrats to Ron Dror, @raphaeljlt, Stephan Eismann, and Masha Karelian at @DrorLab and coauthors at @RDasLab for the front page of @ScienceMagazine
Geometric deep learning of RNA structure https://t.co/qbj61aEKmR
Perspective https://t.co/TKQtfTefLT Podcast https://t.co/6MmrfUPZuk

[Steven Brunton @eigensteve]

New video on the Sparse Identification of Nonlinear Dynamics (SINDy), 5 years later https://t.co/VoiSrosbtc

Machine learning is enabling the discovery of dynamical systems models and governing equations purely from measurement data.

[Miles Cranmer @MilesCranmer]

Our paper demonstrating the power of Bayesian Neural Networks for planetary dynamics comes out in PNAS today!

https://t.co/uc9rIAK3FH (open access)

This paper explores a match made in heaven: chaotic systems and Bayesian neural networks.

Thread:

[Chris Olah @ch402]

Tristan Needham's "Visual Complex Analysis" is probably the best piece of technical exposition I've ever read.

I just learned that he published a new book last year, taking a visual approach to differential geometry. I'm so excited to receive my copy!