Gabriel Mongaras

Some guy trying to teach rocks how to do things.
I also post videos

[Gabriel Mongaras @gmongaras]

But basically, if you take (qk)^2 instead of exp(qk), the inner product is performed in a space with the same (r 2) dim. The main difference is the paper a wedge product is used while squaring uses a dot product. I am very curious if the operations result in accuracy differences

[Gabriel Mongaras @gmongaras]

https://t.co/LfAtMRdtpl
Very interesting paper. It seems like the Grassman attn they develop is very similar to squaring the QK inner product + an input convolution where each head has a different stride. If I get a chance, I wanna compare this to something I've been working on.

[Gabriel Mongaras @gmongaras]

They also have the output gate, which I suspect is a small increase in accuracy if at all
...

[Gabriel Mongaras @gmongaras]

(can't edit posts)
My bad, I meant I didn't include the subtraction on the backward pass which means the gradient was ever so slightly off, but just enough to mess things up after like 60K steps.

[Gabriel Mongaras @gmongaras]

Two lines really killed my Triton kernel on large outputs lol. Interestingly, the error was the same on my test case but not using the max during the forward pass consistently killed the kernel after a while. Very annoying bug as I had to retrain to see if it was fixed.

[Gabriel Mongaras @gmongaras]

Why doesn't OpenAI just add -inf to the logit corresponding to em-dash? Kinda sus

[Gabriel Mongaras @gmongaras]

Ayo dLLM with BERT. I'm surprised it works so well though I know I shouldn't be as dLLM training and BERT training are similar. Makes me wonder if techniques used in encoder models such as BERT, T5, UL2, etc. can also be used to improve diffusion models.
https://t.co/rK8cdALPTJ

[Gabriel Mongaras @gmongaras]

So like it's obviously far from being a brand new type of linear attention, just another step in the evolution of current linear attention approaches. Curious where it goes from here.

[Gabriel Mongaras @gmongaras]

Big fan of linear attention catching on more with Kimi Delta Attn, but most sources I've read make it seem like it's a brand new type of linear attn, however it's basically a single change to gated deltanet plus an efficient way to do the training.
https://t.co/L6EBt70v7t
...

[Gabriel Mongaras @gmongaras]

Adding onto this, gated deltanet is basically deltanet + mamba (looking at the two major papers) and most linear attn blocks follow mamba 2's block design. A lot of modern linear attention follow the mamba 2 algorithm and block design. I am still surprised it got rejected.
...

[Gabriel Mongaras @gmongaras]

I know what I'm reading for the holidays. If there's anything you wanted to know about diffusion models, I'm betting it's in here.
https://t.co/6vjxZjwl2c

[Gabriel Mongaras @gmongaras]

Hype paper! Although the proof is a little limited (assumes real analytic), they basically show transformers are invertible and thus one can reconstruct a prompt in O(num tokens*vocab size) time given the hidden states. I think I'm gonna experiment with this idea a little.

[GLADIA Research Lab @GladiaLab]

LLMs are injective and invertible.

In our new paper, we show that different prompts always map to different embeddings, and this property can be used to recover input tokens from individual embeddings in latent space.

(1/6)

[Gabriel Mongaras @gmongaras]

Feels like there should be a better way to reduce exposure bias than with RL. RL just seems like a good enough way to do it with preference data.

[Gabriel Mongaras @gmongaras]

It's probably documented somewhere but it seems like the main thing RL does after SFT is reduce exposure bias. SFT is needed so you aren't sampling garbage from the model and actually have a good starting point to reduce exposure bias from.

[Gabriel Mongaras @gmongaras]

Again, still need to do a full read through but maybe Mamba 3 is fundamentally different as it seems to be able to use things normal linear attn wouldn't like RoPE and complex numbers?

[Gabriel Mongaras @gmongaras]

Ayo mamba 3 is here! Haven't read through the paper yet. Considering Mamba 2 is empirically already quite close to transformer performance, I am very interested in how this does. The claim that it's better than transformers is a little sus, but I still need to read the paper lol.

[JUNDE WU @JundeMorsenWu]

Mamba3 just silently dropped on ICLR🤯

A faster, longer-context, and more scalable LLM architecture than Transformers

A few years ago, some researchers started rethinking sequence modeling from a different angle.
Instead of stacking more attention layers, they went back to an older idea : state-space models, systems that keep an internal state evolving over time.
That became the foundation for Mamba.

The early versions were promising.
Mamba-1 used continuous-time dynamics with selective memory updates, so it could remember efficiently without the heavy cost of attention.
Mamba-2 went further and showed that state-space updates and attention are actually two sides of the same math, which made it run much faster on GPUs while keeping similar performance.

Now Mamba-3 feels like the design finally matured. It refines how the internal state evolves, how it remembers, and how it uses hardware.
The main update lies in switching from a simple Euler step to a trapezoidal integration, which takes into account both the start and end of each time interval.
That small change makes its memory smoother and more stable over long sequences.
It also lets the hidden state move in the complex plane, which adds a kind of rhythmic, oscillating memory. Instead of just decaying over time, the model can now represent repeating or periodic patterns, the kind of structure language and music often have.
And with a new multi-input-multi-output design, Mamba-3 can process several streams in parallel, making much better use of modern GPUs.

In practice, Mamba-3 opens up a lot of possibilities.
Its ability to handle long sequences efficiently makes it a strong fit for tasks like long-document understanding, scientific time-series, or genome modeling: areas where Transformers struggle with context limits.
Because it runs in linear time and keeps latency stable, it’s also well-suited for real-time applications like chat assistants, translation, and speech interfaces, where responsiveness matters more than raw scale.
And its hardware-friendly design makes Mamba-3 could eventually power on-device or edge AI systems, running large models locally without depending on the cloud.

It’s the kind of architecture that quietly expands from large-context reasoning on servers to lightweight intelligence on everyday devices

[Gabriel Mongaras @gmongaras]

Gated Deltanet and TTT are also linear attention with some extra pieces that make it a better, though the basis is basically the same.
...