Ben Anson

[Ben Anson @benaibean]

@borisdayma i didn't know what Kron PSGD was - this helps slightly, but the details seem very complex

https://t.co/YMBm4JBiLm

[Ben Anson @benaibean]

@chrmanning any intuition on whether you can skip the transformer training component, and instead take an existing model and somehow compress query + KV cache --> m_t ?

I guess the transformer decoder is still very helpful, since without it it'd be hard to guarantee that m_t is predictive

[Xidulu @xidulu]

1/ Sharing a new, interesting project I did during my internship at Microsoft AI Frontiers w/ @JohnCLangford

TL;DR: At decoding time, we feed **previous hidden state** into the input together with token embedding, and it boosts performance for free.

https://t.co/za3cZ4UwWV

[Sam Bowyer @sambowyer__]

My @cohere internship project w/ @kroscoo and @acyr_l is on arXiv! We show that efficient benchmarking (predicting scores from a subset of questions) can be greatly improved using standard feature-selection & regression techniques (mRMR and kernel ridge)! https://t.co/ZVX3lK8N68

[Ben Anson @benaibean]

@zhaisf have you seen https://t.co/l92RoYY2dE ?

it proposes a similar idea, but applied to the residual rather than the values. though I was not able to reproduce much benefit in a language modelling setting

[Ben Anson @benaibean]

@giffmana thank you!

[Ben Anson @benaibean]

Is it possible to _derive_ an attention scheme with effective zero-shot generalisation? The answer turns out to be yes! To achieve this, we began by thinking about desirable properties for attention over long contexts, and we distilled 2 key conditions:

[Ben Anson @benaibean]

@fedzbar thank you!

[Ben Anson @benaibean]

Check out the full preprint on arXiv! https://t.co/yhNG1p4qAg

[Ben Anson @benaibean]

We tested our scale-invariant attention on zero-shot length generalisation tasks (4k -> 64k) and find that it outperforms baselines like LogN and ALiBi. For example, when pretraining:

[Ben Anson @benaibean]

Check out the full preprint on arXiv! https://t.co/yhNG1p4qAg

[Ben Anson @benaibean]

Is it possible to _derive_ an attention scheme with effective zero-shot generalisation? The answer turns out to be yes! To achieve this, we began by thinking about desirable properties for attention over long contexts, and we distilled 2 key conditions:

[Ben Anson @benaibean]

We tested our scale-invariant attention on zero-shot length generalisation tasks (4k -> 64k) and find that it outperforms baselines like LogN and ALiBi. For example, when pretraining: