How much does a language model forget when finetuned on new tasks? We show both model size and optimization matter and forgetting can be nearly eliminated with self-generated replay!
https://t.co/Qs9A4n095s
w/@mrtnm@dongkyucho@ShikaiQiu@rumichunara@Pavel_Izmailov 1/8