Yingcong (@LizzieLyc) will be discussing our recent work in the In-Context Learning Workshop at ICML. We study the optimization landscape to characterize the role of Data, Architecture, and Parameter Efficiency. If you are still around, consider attending her talk... (1/6)
Our paper proves that optimizing an attention layer with next token objective discovers Strongly-Connected Components (SCC) of graphs induced by the training data (throwback to Tarjan's seminal algo).