Axel Laborieux

[Nicolas Zucchet @NicolasZucchet]

Processing delays are everywhere in the brain, which creates synchronization issues between neural activity and corresponding learning signals.

How can we still learn despite that?!

We show that prospective neurons may be a robust answer to that problem!

A 🧵 below 👇

[Axel Laborieux @AxelLaborieux]

@TheAhmadOsman Interested!

[Friedemann Zenke @hisspikeness]

We're hiring! Come build models of how the brain learns and simulates a world model. We have several openings at PhD and postdoc levels, including a collab with @georg98keller lab on designing regulatory elements to target distinct neuronal cell types.
https://t.co/DakerWs5Sz

[Yoshua Bengio @Yoshua_Bengio]

As we move towards more powerful AI, it becomes urgent to better understand the risks in a mathematically rigorous and quantifiable way and use that knowledge to mitigate them. More in my latest blog entry where I describe our recent paper on that topic.
https://t.co/emiQxTvWrd

[Matthew Green @matthew_d_green]

So Apple has introduced a new system called “Private Cloud Compute” that allows your phone to offload complex (typically AI) tasks to specialized secure devices in the cloud. I’m still trying to work out what I think about this. So here’s a thread. 1/

[Nicolas Zucchet @NicolasZucchet]

Why do state-space models work so well?

With @orvieto_antonio, we study their learning dynamics and find that diagonal recurrence is key:
1. It helps in better conditioning the loss landscape
2. It facilitates Adam's job by making the Hessian diagonal

📝 https://t.co/7Q7In6WMUU

[Friedemann Zenke @hisspikeness]

@AxelLaborieux and I have been working on an introductory book chapter on synaptic complexity for continual learning. Please check it out! We welcome feedback.

[Friedemann Zenke @hisspikeness]

1/6 Surrogate gradients (SGs) are empirically successful at training spiking neural networks (SNNs). But why do they work so well, and what is their theoretical basis? In our new preprint led by @JuliaGygax4, we provide the answers: https://t.co/QkQ4MniGIG

[Daniel Johnson @_ddjohnson]

Excited to share Penzai, a JAX research toolkit from @GoogleDeepMind for building, editing, and visualizing neural networks! Penzai makes it easy to see model internals and lets you inject custom logic anywhere.

Check it out on GitHub: https://t.co/mas2uiMqj9

[Axel Laborieux @AxelLaborieux]

@Michal_Zajac_ @tuytelaarslab @GMvandeVen Thanks for sharing the nice work!

You might find this paper which does something similar with invertible networks interesting (see Eq. 9):
https://t.co/npKcFND1Oy

[Surya Ganguli @SuryaGanguli]

1/Our new paper "Geometric Dynamics of Signal Propagation Predict Trainability of Transformers" lead by Aditya Cowsik,Tamra Nebabu w/Xiaoliang Qi yields theory experiment match for how token rep geometry evolves thru transformers, reveals two phase transitions and 4 phases and..

[Georg Keller @georg98keller]

Predictive Processing: A Circuit Approach to Psychosis - https://t.co/xw0x2gNqli - very excited to see this out - a great collaboration with @PhilippSterzer. These ideas will certainly guide my research for the coming years.

[Sebastian Goldt @sebastiangoldt]

🚨Summer school alert 🚨Want to learn more about statistical physics applied to machine learning 🤖and neuroscience🧠? Then come join us at beautiful Lake Como for lectures by great Florent @KrzakalaF, Matthieu Wyart, and Francesca Mastrogiuseppe, as well as seminars by... 👇1/2

[Yishan @yishan]

Google’s Gemini issue is not really about woke/DEI, and everyone who is obsessing over it has failed to notice the much, MUCH bigger problem that it represents.

(1/n)

[Pedro Domingos @pmddomingos]

If you’re an AI grad student, don’t try to invent the next GPT. Invent the next backprop.

[Nicolas Zucchet @NicolasZucchet]

How do modern RNNs/SSMs such as Mamba perform on in-context learning tasks? How do they relate to attention-based models like Transformers?

We find that modern RNNs can implement attention and that they leverage it to solve ICL tasks in an attention-based manner!

(1/6)

[Damien Querlioz @DamienQuerlioz]

Our self-powered memristor neural network! It harvests energy with a tiny solar cell, and adjusts its accuracy depending on available energy! Very proud of this one @IM2NP_CNRS @C2N_com @CEA_Leti @IPVF_institute @NatureComms🌞 1/2

[Claire MB @Clairemb1]

*Now on biorxiv!* How does excitatory-inhibitory connectivity affect the geometry of neural population responses to odors? Work with @RainerFriedri12 and @hisspikeness at @FMIscience https://t.co/80AeG48UsA

[Nicolas Zucchet @NicolasZucchet]

Excited to announce that our paper "Online learning of long-range dependencies" was accepted #NeurIPS2023!
Our learning rule enables efficient long-term credit assignment in recurrent neural networks while being purely online!

A short 🧵

[Nicolas Zucchet @NicolasZucchet]

Ever wondered how to implement weight-transport-free deep learning algorithms in JAX/Flax?
Happy to announce bioflax https://t.co/PFu0gtt3wN, a library developed by @yaschimpf and me that simply implements those algorithms.
A quick 🧵on why JAX and how our code works👇