Shreyas Kadekodi

CSE phd student @ UC San Diego

[Sean O'Brien @seano_research]

(1/n)

32 extra parameters → 3.4% saved pretraining compute

RoPE’s rotation frequencies matter, but they’re usually fixed at hand-picked values. In a new work with @kpetro_research, we let the network decide: LeRoPE learns the RoPE frequencies during pretraining.

Thread below!