Ari Benjamin

AI for neuroscience. @aribenjamin elsewhere

[Ari Benjamin @arisbenjamin]

@BoWang87 What are your thoughts on the apparent poor performance of single-cell foundation models on predicting the effects of genetic perturbations?

e.g.
https://t.co/dMLoKZCZoh and
https://t.co/BBUbxMseJI

[Ari Benjamin @arisbenjamin]

@elonmusk there's nothing 'free speech absolutist' in defunding scientists based on keywords in their grants

[Ari Benjamin @arisbenjamin]

So

[Ari Benjamin @arisbenjamin]

Lovely to see good discussion on X again.

I love the spectral theory of model/brain alignment paper mentioned in this disagreement. But what does it imply? I disagree with the conclusion – but I agree that we should grapple with the question. 🙏🏻 for elevating it. (1/🧵)

[Rylan Schaeffer @RylanSchaeffer]

I wanted to wait before responding to @aran_nayebi 's post (https://t.co/OVon9gIKrF) because it seemed like tempers were high, and to steal a phrase I learned this week from a friend:

The goal is to create light 💡 not heat 🔥

- To briefly restate our overarching perspective, comparing a system and a model requires specifying what similarity to that system means
- Only once “neural similarity” has been defined can then one propose proxy metrics for neural similarity (e.g., test R^2 of linear regression)
- However, as the field optimizes against a fixed proxy metric, the field begins Goodharting: overfitting to the proxy at the cost of true “neural similarity” (which we believe is a neural system-, setting- and task- dependent notion)

Our critique of neural comparisons as a methodology used today is neural similarity has been inadequately defined and the field has devolved into overfitting to neural predictivity

While we appreciate the interest, Dr. Nayebi misunderstands or misrepresents both our work and prior work. Let's explain why.

1/7

[Ari Benjamin @arisbenjamin]

To me, two representations are similar when they can be used similarly. We cannot forget the learnability of readouts. A~B when, for task C, A->C and B->C have similar sample complexity.

Regressing A<->B measures usability in a task-agnostic way.

[Ari Benjamin @arisbenjamin]

We developed this framework to think about catastrophic forgetting. But it seems like a generally useful interpretation. What could you do with this insight?

[Ari Benjamin @arisbenjamin]

Neural networks are ensembles. Really! In new work, we show that a network with N edges can be seen as a Mixture of Experts model with N experts. Even the input edges contribute an expert. https://t.co/v5hVy3DtCi.

With @ChristianPehle and @darsnack.

[Kording Lab 🦖 @KordingLab]

Regular reminder. As an academic, student, postdoc, or professor, you accept less pay and more stress. It is only worth doing if what you do truly matters. Do that high-risk project. Because the boring project does not justify the downsides of academia. For the love of ideas.

[Ari Benjamin @arisbenjamin]

@helenxhou Congrats @darsnack and team!

[Ari Benjamin @arisbenjamin]

Endorsed! I attended this course last year in London. If you use ANNs as models of the brain, this course will teach you to go deeper. What's happening inside those networks and why?

[SueYeon Chung @s_y_chung]

Excited to announce this summer school on analytical approaches (eg statistical physics and probability theory) for understanding deep learning models and higher-level cognition in NYC!

Analytical Connectionism Summer School (Aug 12-23, '24)

Speakers:
Jonathan Cohen
@CPehlevan
Linda Smith
@kchonyc
André Fenton
@adelegoldberg1
@EeroSimoncelli
@EngelTatiana
@chklovskii

@FlatironCCN @FlatironInst @NYUDataScience @NYU_CNS @GatsbyUCL

[Ari Benjamin @arisbenjamin]

@CamilleTestard @Nature yay congrats to you and the whole team!

[Ari Benjamin @arisbenjamin]

Theoretical neuroscience should pay attention to these developments.
Learning does not need full-rank gradients; low-rank approximations work quite well. What different learning circuits might this inspire? Maybe an extracortical loop into BG/thalamus? ;)

[AK @_akhaliq]

GaLore

Memory-Efficient LLM Training by Gradient Low-Rank Projection

Training Large Language Models (LLMs) presents significant memory challenges, predominantly due to the growing size of weights and optimizer states. Common memory-reduction approaches, such as low-rank

[Ari Benjamin @arisbenjamin]

@BenjoCowley hoping this stays a series! I want to these hung in a Chelsea gallery

[Tyler Sloan @quorumetrix]

My latest #neuro #visualization:
An intuition pump on connectivity in the #brain
~75K #neurons in the #MICrONS mm³.
I/O segregated by layers, function is columnar; the cortical column is the unit of cortical computation.
A typical #pyramidal cell has over >10K input #synapses.

[Ari Benjamin @arisbenjamin]

(ok, all set. DM me and I can pay it forward)

[Ari Benjamin @arisbenjamin]

Does anyone have an invite for the blue place? DMs welcome :)

[Andy Keller @t_andy_keller]

Traveling waves are known to exist throughout the brain in a variety of forms — there are many hypotheses, but their exact computational role is debated.

Together with @wellingmax we built an RNN which exhibits traveling waves to see what it could do. Here’s what we think:

1/7

[Ari Benjamin @arisbenjamin]

@KordingLab @SternNachi There's a close parallel with some DL theory work in linear nets / deep matrix factorization, like that of Saxe or Arora's labs. could be worth citing those in a revision

[Ari Benjamin @arisbenjamin]

@KordingLab nice to see this out! @SternNachi I remember this being presented in K-Lab. love the parallel of learning theory to physical systems & proteins