Introducing CorText: a framework that fuses brain data directly into a large language model, allowing for interactive neural readout using natural language.
tl;dr: you can now chat with a brain scan 🧠💬
1/n
We're excited to announce we're starting a Journal Club. And our first meeting is scheduled for tomorrow!
@__init_self will present her work, CorText: Brain-Language Fusion Enables Interactive Neural Readout and In-Silico Experimentation
Tomorrow at 10:15am ET, join Discord!
Excited about our new preprint: “The illusory simplicity of the feedforward pass: evidence for the dynamical nature of stimulus encoding along the primate ventral stream”
https://t.co/IbtXBTQb5t
Work with Sushrut Thorat, Anna Mitola, Paolo Papale, Peter König & Tim Kietzmann
@torwager As another example, detailed info such as text in the img does not work! For me, the interesting question here is whether the limitations are in the training of the model (limited samples/annotations), or in the neural data (what content is in principle decodable from it?).
Introducing CorText: a framework that fuses brain data directly into a large language model, allowing for interactive neural readout using natural language.
tl;dr: you can now chat with a brain scan 🧠💬
1/n
@torwager Thanks! The current model can decode most semantic clusters in NSD well, although it sometimes confuses within-category (big animal, but is it a cow or elephant?). It's also quite good at numerosity (e.g., how many people), but struggles more with positional information. 1/2
@yosuke_neuro Thanks for checking out our paper! There is no simulation of neural data in our framework, we embed and feed it to the model as ‘context’ for doing Q&A - like a VLM, but instead of an image its brain data! Generating synthetic neural data is another interesting research line :)
We are convinced that these results mark a shift from static neural decoding toward interactive, generative brain-language interfaces.
Preprint: https://t.co/882OaOre5X
🚨 Finally out in Nature Machine Intelligence!! "Visual representations in the human brain are aligned with large language models"
https://t.co/GB5k6IV4Jg
Our new study in @NatComputSci, led by Haibao Wang, presents a neural code converter aligning brain activity across individuals & scanners without shared stimuli by minimizing content loss, paving the way for scalable decoding and cross-site data analysis. https://t.co/si0qg66Nu9
Exciting new preprint from the lab: “Adopting a human developmental visual diet yields robust, shape-based AI vision”. A most wonderful case where brain inspiration massively improved AI solutions.
Work with @lu_zejin@martisamuser and Radoslaw Cichy
https://t.co/XVYqQPjoTA