Simons Institute for the Theory of Computing

The world's leading venue for collaborative research in theoretical computer science. Follow us at

[Simons Institute for the Theory of Computing @SimonsInstitute]

1/5 Imagine taking an fMRI scan of the brain of a person viewing an image, and reconstructing what the person saw from the fMRI alone. That's Brain-IT. "This is state-of-the-art image decoding from fMRI," said Michal Irani (@WeizmannScience), at the Simons Institute.

[Simons Institute for the Theory of Computing @SimonsInstitute]

"Of course, we also have failures," said Michal Irani, at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning. For e.g., the fMRI scan of a person viewing a cat became a bear. Video: https://t.co/vOxl0SsAlq
Paper: https://t.co/5wo4LvPoVD

[Simons Institute for the Theory of Computing @SimonsInstitute]

1/5 Imagine taking an fMRI scan of the brain of a person viewing an image, and reconstructing what the person saw from the fMRI alone. That's Brain-IT. "This is state-of-the-art image decoding from fMRI," said Michal Irani (@WeizmannScience), at the Simons Institute.

[Simons Institute for the Theory of Computing @SimonsInstitute]

4/5 Here are some examples of successful reconstructions of the images observed by a person, using only the fMRI images. For each pair, left is the original image, right is the reconstruction from the fMRI scan. Paper: https://t.co/5wo4LvPoVD Roman Beliy et al.

[Simons Institute for the Theory of Computing @SimonsInstitute]

3/3 "This is part of a larger...collaborative effort to build foundation models," said Eva Dyer of @Penn at the Simons Institute. Such models would unify diverse neural data with varying temporospatial resolutions, from multiple species and tasks. Video: https://t.co/9JYqIPIVCe

[Simons Institute for the Theory of Computing @SimonsInstitute]

1/3 From task-specific neural data to foundation models. In neuroscience, studying the neural activity of some behavior in an animal only gives us "a snapshot of the full activity that might be present across the brain," said Eva Dyer of @Penn at the Simons Institute.

[Simons Institute for the Theory of Computing @SimonsInstitute]

2/3 "We've been excited by the idea of taking...fragmented neural datasets & putting them into one unified model...[that] is greater than the sum of its parts," said Eva Dyer of @Penn at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning

[Simons Institute for the Theory of Computing @SimonsInstitute]

During interactions with others, "there are synchronizations that appear between two nervous systems...we have this phenomenon within brains. This [also] happens between agents," said Guillaume Dumas (@introspection) of @Mila_Quebec at the Simons Institute.

[Simons Institute for the Theory of Computing @SimonsInstitute]

The Dark Matter of Neuroscience. For decades, "neuroscience didn't have much to say about two people...in a dynamical interaction," said Guillaume Dumas (@introspection) of @Mila_Quebec at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning

[Simons Institute for the Theory of Computing @SimonsInstitute]

"When we are interacting with others there is a dynamics that transcends the...people that are participating, and these dynamics in return also shape the people," said Guillaume Dumas (@introspection) of @Mila_Quebec at the Simons Institute. Video: https://t.co/CcECZSi8lT

[Simons Institute for the Theory of Computing @SimonsInstitute]

"What we foresee in...cognitive science is that we move progressively from 'others' as a problem to 'others' as a form of affordances for interactions, enabling things that we aren't able to do on our own." said Guillaume Dumas of @Mila_Quebec, at the Simons Institute.

[Simons Institute for the Theory of Computing @SimonsInstitute]

The African philosophy Ubuntu — "I am because you are" — inspires Guillaume Dumas (@introspection) of @Mila_Quebec, in his research on social cognition and interaction. Dumas spoke at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning

[Simons Institute for the Theory of Computing @SimonsInstitute]

"The question of 'other' and social cognition is kind of central in cognitive science," said Guillaume Dumas (@introspection) of @Mila_Quebec, at the Simons Institute. Video: https://t.co/CcECZSi8lT

[Simons Institute for the Theory of Computing @SimonsInstitute]

1/3 For AIs to be socially intelligent, they'll have to learn internal "social world models" of people & social interactions, not just physical world models, said Paul Liang (@MITEECS) at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning

[Simons Institute for the Theory of Computing @SimonsInstitute]

3/3 Paul Liang of @MITEECS spoke of his lab's work developing socially-intelligent AI, focusing on modeling touch, olfaction, and the internal states that drive human behaviors, at the Simons Institute. Video: https://t.co/JIhJHIBwZ4

[Simons Institute for the Theory of Computing @SimonsInstitute]

1/3 For AIs to be socially intelligent, they'll have to learn internal "social world models" of people & social interactions, not just physical world models, said Paul Liang (@MITEECS) at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning

[Simons Institute for the Theory of Computing @SimonsInstitute]

2/3 Such an AI would be "capable of understanding the environment, understanding [and] interacting with people, eventually building towards [a] social world model," said @pliang279 at the Simons Institute workshop on Topics in Intelligence: World Models and Social Reasoning

[Simons Institute for the Theory of Computing @SimonsInstitute]

The schedule of talks for our Aug. 24–28 ICM Satellite Conference on Spectral Theory, High-Dimensional Expansion, and Pseudorandomness is now online:

https://t.co/ftalbgXILK