@michaeltgao@constructor_agi Looks like a marginal improvement over existing sparse attention works (without comparing against any of them!)
> we have not run the sparse-attention methods above under the same training
Excluding RoPE from the retrieval vectors is a clever way to improve extrapolation though
Yeah I was exploring something similar and found that a diagonal shampoo preconditioner based on the instantaneous gradients works well at low cost for square states.
The tricky part is that the sqrt of the preconditioner should be applied to the noise, which can mess with the SNR (but skipping that is equivalent to adaptive tempering)
@roicohen9 Under that interpretation I'd be curious to see the meta-model predict prefix-tuning adapters that get inserted into the KV cache (instead of LoRA adapters).
Then you get an architecture equivalent to T5 Gemma 2: https://t.co/FtGXcWNO5u
@roicohen9 I interpret this less as meta-learning and more as a classic T5-style encoder-decoder architecture with:
1. a small encoder
2. linear cross attention instead of softmax
3. the linear attention mechanism parallel to weight matrices so it's equivalent to a LoRA
@jayden_teoh_ 3. Use a generative prediction loss (like with a flow-matching head [R1]) instead of regression losses. This would let help the model encode belief states non-linearly (recent evidence shows this is more natural [R2]).
R1: https://t.co/1K81TAjh44
R2: https://t.co/KEsuqU4x2R
@jayden_teoh_ 2. Use an intermediate hidden state as the target latent. Intermediate states often have richer and more useful representations than final ones: https://t.co/QnZgXK9iRi
@huskydogewoof Or maybe you could get the best of both worlds by calculating latent states in parallel using a privileged encoder: https://t.co/S6Ki83qq3K
@huskydogewoof Someone needs to train a latent feedback model using full serial BPTT instead of fixed-point iterations.
It would be slow, but you'd get much deeper credit assignment and incentivize intermediate reasoning states.
Maybe you could do continued pretraining after a fixed-point run
@xingyudang related:
https://t.co/qSoOOgHj0v - langevin inner loops induce generalization. Ignoring normalization, T^3 is a special case with L=1 and no added noise (but SGD steps are still noisy on their own)
@xingyudang Interesting that it only improves performance when tokens repeat. This makes me think it's acting more like a regularizer than anything.
Maybe optimizing the weights to be robust under noisy mini-batches pushes towards wide minima