Mathis Pink

👀🧠(x) | x ∈ {👀🧠,🤖}
PhD student @mpi_sws_
trying to trick rocks into thinking and remembering.

[Mathis Pink @MathisPink]

1/n🤖🧠 New paper alert!📢 In "Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks" (https://t.co/S8BZzkFVM6) we introduce SORT as the first method to evaluate episodic memory in large language models. Read on to find out what we discovered!🧵

[Abhilasha Ravichander @lasha_nlp]

How can generative AI better support human creativity, without limiting it?

If you have thoughts, we invite submissions to our ICML workshop on Generative AI, Creativity, and Human-AI Co-Creation (@creativeai_ws)

📍 July 2026, Seoul
📄 Submit by: April 24 (AOE)

⏬ (1/2)

[Abhilasha Ravichander @lasha_nlp]

📢 I'm recruiting PhD students at MPI!!

Topics include:
1⃣ LLM factuality, reliable info synthesis and reasoning, personalization + applications in real-world inc. education, science
2⃣ Data-centric interpretability
3⃣Creativity in AI, esp scientific applications 🧵1/2

[Tim Kietzmann @TimKietzmann]

Exciting new preprint from the lab: “Adopting a human developmental visual diet yields robust, shape-based AI vision”. A most wonderful case where brain inspiration massively improved AI solutions.

Work with @lu_zejin @martisamuser and Radoslaw Cichy

https://t.co/XVYqQPjoTA

[Omer Moussa @ohmoussa2]

🚨Excited to share our latest work published at Interspeech 2025: “Brain-tuned Speech Models Better Reflect Speech Processing Stages in the Brain”! 🧠🎧
https://t.co/LeCs6YfbZp
W/ @mtoneva1
We fine-tuned speech models directly with brain fMRI data, making them more brain-like.🧵

[Martina Vilas @martinagvilas]

We will be presenting this 💫 spotlight 💫 paper at #ICLR2025. Come say hi or DM me if you're interested in discussing AI #interpretability in Singapore!

📆 Poster Session 4 (#530)
🕰️ Fri 25 Apr. 3:00-5:30 PM
📝 https://t.co/xSyTt7bqxy
📊 https://t.co/wftoGjLZrd

[Sebastian Michelmann @s_michelmann]

Excited to share our new preprint https://t.co/c8kgodk5hS with @mtoneva1, @ptoncompmemlab, and @manojneuro), in which we ask if GPT-3 (a large language model) can segment narratives into meaningful events similarly to humans. We use an unconventional approach: ⬇️

[Karan Goel @krandiash]

A few interesting challenges in extending context windows.

A model with a big prompt =/= "infinite context" in my mind. 10M tokens of context is not exactly on the path to infinite context.

Instead, it requires a streaming model that has
- an efficient state with fast incremental state updates
- strong compression & abstraction of multimodal history
- effective at reasoning over state to generate future action

(The model needs to be able to run forever.)

I dislike the phrase "infinite context" as a matter of taste, since it suggests an extension of the retrieval based paradigm where you reason over context windows in order to answer a query. That's not quite what's going on.

Instead, memory is about building useful abstraction over histories. It's nice to think of it as a state with a choice of method for storing/updating information -- this is more in line with a model that always runs and streams in information. Transformers / SSMs are just different ways to choose how to model the state and update it. There are more abstractions left to uncover.

There are also new data regimes and training algorithms required to produce these kind of long-term memory models. Interaction is a core part of the way these models operate.

We don't seem to have the right evals that measure useful proxies that are in line with this notion of memory. Ideally, it's not just about being able to remember facts -- the model should get better at retaining skills (compound knowledge), and learning new skills over time (learning to learn faster).

This is a very interesting direction that I'm personally super excited to be working on.

[François Chollet @fchollet]

Anyway, glad to see that the whole "let's just pretrain a bigger LLM" paradigm is dead. Model size is stagnating or even decreasing, while researchers are now looking at the right problems -- either test-time training or neurosymbolic approaches like test-time search, program synthesis, and symbolic tool use.

[Mathis Pink @MathisPink]

@goodside https://t.co/S8BZzkFVM6

[Mathis Pink @MathisPink]

@goodside LLMs currently lack episodic memory (EM), which is a form of long-term memory that is different from semantic memory in that it is about single specific encountered sequences. In a recent paper we created a benchmark (SORT) that evaluates temporal order memory as a proxy for EM!

[Mathis Pink @MathisPink]

https://t.co/rahFgZKfH7 We think this is because LLMs do not have parametric episodic memory (as opposed to semantic memory)! We recently created SORT, a new benchmark task that tests temporal order memory in LLMs

[Riley Goodside @goodside]

An LLM knows every work of Shakespeare but can’t say which it read first. In this material sense a model hasn’t read at all.

To read is to think. Only at inference is there space for serendipitous inspiration, which is why LLMs have so little of it to show for all they’ve seen.

[François Fleuret @francoisfleuret]

Consider the prompt X="Describe a beautiful house."

We can consider two processes to generate the answer Y:

(A) sample P(Y | X) or,

(B) sample an image Z with a conditional image density model P(Z | X) and then sample P(Y | Z).

1/3

[Mathis Pink @MathisPink]

4/n💡We find that fine-tuning or RAG do not support episodic memory capabilities well (yet). In-context presentation supports some episodic memory capabilities but at high costs and insufficient length-generalization, making it a bad candidate for episodic memory!

[Omer Moussa @ohmoussa2]

We are so excited to share the first work that demonstrates consistent downstream improvements for language tasks after fine-tuning with brain data!!
Improving semantic understanding in speech language models via brain-tuning
https://t.co/HVAzk36Wga

W/ @dklakow, @mtoneva1

[Mathis Pink @MathisPink]

@valentina__py thanks Valentina!

[Mathis Pink @MathisPink]

1/n🤖🧠 New paper alert!📢 In "Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks" (https://t.co/S8BZzkFVM6) we introduce SORT as the first method to evaluate episodic memory in large language models. Read on to find out what we discovered!🧵

[Mathis Pink @MathisPink]

@vvobot @moon91007207 @javiturek @s_michelmann @alex_ander @mtoneva1 12/n Check out our full paper, SORT evaluation code to test your own models, and Book-SORT dataset on 🤗-datasets:
Paper: https://t.co/S8BZzkFVM6
Code: https://t.co/RkfQFhlO0Z
Book-SORT: https://t.co/LIQDgAMvr7

[Mathis Pink @MathisPink]

11/n SORT is the product of joint work together with @vvobot, @moon91007207, Jianing Mu, @javiturek, Uri Hasson, Ken Norman, @s_michelmann, @alex_ander and @mtoneva1