Anand Gopalakrishnan

Postdoc at @Harvard with @du_yilun and @gershbrain. PhD with @SchmidhuberAI. Previously: Apple MLR, AWS AI Lab. 7\. Same handle on 🦋

[Anand Gopalakrishnan @agopal42]

Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information.
Our fix (PoPE) is simple but powerful. Paper: https://t.co/XlltfcSwHQ

[Anand Gopalakrishnan @agopal42]

@liu_yuejiang @NUSComputing Congrats! 🎉

[Anand Gopalakrishnan @agopal42]

@xuanalogue Yeah, I agree. I guess we'll all be have to really internalise the idea of doing something for its own sake, which is a good outcome.

[Anand Gopalakrishnan @agopal42]

@xuanalogue So the only way for society to deal with the effects of large scale automation in the long run is to strive to reach enlightenment. So win-win overall? :D

[Anand Gopalakrishnan @agopal42]

@LouisKirschAI @SchmidhuberAI @inherent_labs Congratulations!! 🎉

[Luca Ambrogioni @LucaAmb]

1/?) As promised to Sander Dieleman (@sedielem), we’re finally excited to share:

Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion

We show that continuous diffusion can achieve very strong language modeling performance when operating directly on bitstreams, outperforming masked and uniform diffusion baselines, and essentially matching autoregressive models under our evaluation settings.

[Michael C. Mozer @mc_mozer]

[1/5] Intelligent behavior relies on maintaining an evolving, dynamic model of the environment. But even frontier models sometimes fail at tracking state in dialogs, e.g., actual transcript from 4/20/2026:

[Anand Gopalakrishnan @agopal42]

@mli0603 Nice blogpost and explanations! You might find our recent work helpful in understanding another failure mode of RoPE -- entanglement of 'what' and 'where'. https://t.co/STVBkKY58J

[Anand Gopalakrishnan @agopal42]

Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information.
Our fix (PoPE) is simple but powerful. Paper: https://t.co/XlltfcSwHQ

[Kazuki Irie @kzkirie]

Back in 2019, I reduced transformer LM KV-cache size by: (1) setting K=V (storing only K), (2) deeper FF blocks & fewer self-attn layers overall.

Published at ICASSP 2020. To my knowledge, the first publication on KV-cache reduction--lmk if you know anything older!

[Jonas @LoosJonas]

Can we replace RoPE with PoPE (Polar Coordinate Positional Embeddings) in pretrained language models?

Turns out we can! Using small pythia models, after a small recalibration (~2% of pretrain), we get significantly better length generalization.

1/3

https://t.co/Jf397xTsU9

[Hansen Lillemark @hansenlillemark]

State of the art World Models still lack a unified world memory for representing and predicting dynamics out of their field of view.

Why is that, and how can we fix it?

Introducing Flow Equivariant World Models: models with memory capable of predicting out of view dynamics!🧵⬇️

[Andy Keller @t_andy_keller]

When you're crossing the street and turn your head, you typically remember whether or not a car is coming from the other direction - so why can't today's world models?

Introducing Flow Equivariant World Models
https://t.co/xUmWLjj0cW

Led by @hansenlillemark & @huskydogewoof🧵👇

[Anand Gopalakrishnan @agopal42]

@sasuke___420 @jm_alexia No we don't. By partial RoPE you mean applying rotations on a subset of all channels/features? Since that's an orthogonal design choice and can be done on both RoPE and PoPE we decided to compare the simplest versions.

[Kazuki Irie @kzkirie]

Humans can't write programs that classify cats vs dogs. Deep learning lets GD write that program.

Continual learning is the same: it's too hard for us to design good CL algorithms. Let GD write that algorithm too. That’s the idea of metalearning CL algos: https://t.co/FjmEgwDlaO

[Anand Gopalakrishnan @agopal42]

@francoisfleuret To _err is human, to _errr is undefined

[Anand Gopalakrishnan @agopal42]

@deaton_jon Yes that's true, but the eqns (7-10) were presented in a feature/channel-wise manner. So we wrote an extra multiplication (per channel). Thanks for your response!

[Anand Gopalakrishnan @agopal42]

Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information.
Our fix (PoPE) is simple but powerful. Paper: https://t.co/XlltfcSwHQ

[Anand Gopalakrishnan @agopal42]

@eshear By log-polar do you mean log spaced frequencies (thetas) used in RoPE? Or something else?

[Anand Gopalakrishnan @agopal42]

@imayank42 Plan to release the repo soon-ish. Thanks for your interest!

[Anand Gopalakrishnan @agopal42]

Posted this a day early and the pun practically writes itself. Noooooo!

[Anand Gopalakrishnan @agopal42]

Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information.
Our fix (PoPE) is simple but powerful. Paper: https://t.co/XlltfcSwHQ

[Anand Gopalakrishnan @agopal42]

11/ Key takeaway: The what-where entanglement in RoPE hurts sequence modelling performance and length generalization.

PoPE's disentanglement provides a powerful inductive bias that solves these issues.

Huge thanks to my co-authors @robert_csordas , @SchmidhuberAI, @mc_mozer !