Neehal Tumma

ML PhD @MIT researching efficient architectures | @LiquidAI | Prev CS/Math @Harvard

[Neehal Tumma @ntumm120]

Some say Gated DeltaNet > Mamba-2. Others say Mamba-2 > Gated DeltaNet.

But what if Gated DeltaNet = Mamba-2? 👀

Well maybe not exactly — but with least-squares preconditioning, we show that they reduce to the same recurrence!

We use this lens to design PDN, PGDN, and PKDA: preconditioned delta-style recurrences that outperform their unpreconditioned counterparts at scale 📈

📄 Paper: https://t.co/i3A3klD8Sa w/ @loo_noel @liquidai
💻 Code: https://t.co/8EbdSjIogo

[Neehal Tumma @ntumm120]

@loo_noel and I will be presenting Preconditioned DeltaNet tomorrow at ICML! Stop by if you wanna chat about recurrences, test-time training or LeBron’s free agency :)

Poster session: Hall A #2915 at 10:30am, July 7th

[Neehal Tumma @ntumm120]

Some say Gated DeltaNet > Mamba-2. Others say Mamba-2 > Gated DeltaNet.

But what if Gated DeltaNet = Mamba-2? 👀

Well maybe not exactly — but with least-squares preconditioning, we show that they reduce to the same recurrence!

We use this lens to design PDN, PGDN, and PKDA: preconditioned delta-style recurrences that outperform their unpreconditioned counterparts at scale 📈

📄 Paper: https://t.co/i3A3klD8Sa w/ @loo_noel @liquidai
💻 Code: https://t.co/8EbdSjIogo

[Sajad Movahedi @Sajad_Movahedi_]

Looped models scale compute by iterating in depth, demonstrating great potential in reasoning. But when should the looping stop, and how to keep it trainable at tens of thousands of unrolled layers? FPRM solves both. (1/7)

[Neehal Tumma @ntumm120]

@osieberling @liquidai Yeah this is a clean interpretation, agreed dynamic conv is more expressive. Curious if that added expressivity buys you anything on language over gated static though

[Neehal Tumma @ntumm120]

Cool paper showing the utility of data-dependent short convolutions.

One comparison I’d love to see: explicit filter conditioning used here vs the LFM style approach from @liquidai, where the short conv kernel is static but the local operator remains input-dependent via gates.

Basically dynamic vs gated static FIR filters.

[Rosinality @rosinality]

https://t.co/jJWPgDd1bg

Dynamic convolution on QKV! An old idea came back again.

[Neehal Tumma @ntumm120]

@rshia_afz @tilderesearch Also general DPLR would be interesting

[Mihir Bafna @mihirbafna14]

Excited to share SwitchCraft, our first step toward designing complex protein function through state switching (ligand responsive binders, allosteric control, biosensors, and more).

Stay tuned for wet-lab results from our amazing collaborators very soon!

[Neehal Tumma @ntumm120]

@jeff_rey_chiu Can Anthropic give me some compute?

[Neehal Tumma @ntumm120]

Some say Gated DeltaNet > Mamba-2. Others say Mamba-2 > Gated DeltaNet.

But what if Gated DeltaNet = Mamba-2? 👀

Well maybe not exactly — but with least-squares preconditioning, we show that they reduce to the same recurrence!

We use this lens to design PDN, PGDN, and PKDA: preconditioned delta-style recurrences that outperform their unpreconditioned counterparts at scale 📈

📄 Paper: https://t.co/i3A3klD8Sa w/ @loo_noel @liquidai
💻 Code: https://t.co/8EbdSjIogo

[Neehal Tumma @ntumm120]

@YifeiZuoX Agreed, I think that would be a super interesting way to push beyond soft max attention. Also Ive read your LLA paper, cool stuff!

[Neehal Tumma @ntumm120]

@VaibhavBerlia Yeah we looked into this a bit, Woodbury solve is expensive though

[Neehal Tumma @ntumm120]

@skippperD Means a lot man 😂

[Neehal Tumma @ntumm120]

(14/N) Next up: we’re working on updating our kernels for the latest flash-linear-attention stack, including the newer GDN Tilelang and FlashKDA kernels, so stay tuned!

And if you made it this far, thanks for reading :)

[Neehal Tumma @ntumm120]

(13/N) But we think this only scratches the surface.

PDN/PGDN/PKDA are just three points in a larger design space: different recurrence families + different preconditioners.

There is a huge optimization literature on preconditioning that can be translated into recurrence design.

The bigger message: preconditioning is a useful new lever for designing linear recurrences.