Top Tweets for #DenseFormer

over 2 years ago

We propose #DenseFormer, a simple modification that performs—after each transformer block—a weighted average of past representations. We call those operations “Depth Weighted Average” (DWA). We learn the DWA weights during training. 2/🧵

MatPagliardini's tweet photo. We propose #DenseFormer, a simple modification that performs—after each transformer block—a weighted average of past representations. We call those operations “Depth Weighted Average” (DWA). We learn the DWA weights during training.
2/🧵 https://t.co/y9rsGCPHTs

Matteo Pagliardini @MatPagliardini

over 2 years ago

A tweak in the architecture of #Transformers can significantly boost accuracy! With direct access to all previous blocks’ outputs, a 48-block #DenseFormer outperforms a 72-block Transformer, with faster inference! A work with @akmohtashami_a,@francoisfleuret, Martin Jaggi. 1/🧵

MatPagliardini's tweet photo. A tweak in the architecture of #Transformers can significantly boost accuracy!

With direct access to all previous blocks’ outputs, a 48-block #DenseFormer outperforms a 72-block Transformer, with faster inference!

A work with @akmohtashami_a,@francoisfleuret, Martin Jaggi.
1/🧵 https://t.co/iumzIoX8Ou

165

836

207K

Top Tweets for #DenseFormer

Last Seen Hashtags on Sotwe

Trends for you

Most Popular Users