Zihan Shao

[Siyu Chen @siyuc3141]

Looped transformers often save compute by truncating backpropagation to the last few loops.

Is it worth it? We trained ~600 models to find out 👀

Short answer: only if the loop count is fixed. Otherwise, run fewer loops and train them all!

📄 https://t.co/4vPIXEUdDJ
🧵👇