新しいフェーズに入ったなぁ.
2025年は長く記憶されることになると思っていたが,それに続く2026年も数学にとって歴史的な年になるのかもしれない.生きてこんな歴史上の転換点にめぐり合うことになろうとは.
Ten advances in mathematics and theoretical computer science
https://t.co/RBfXWINcyq
transformers are almost surely injective, meaning, you can recover the exact input text from hidden states
parameter settings where two different prompts land on the same hidden state form a measure-zero set under Lebesgue measure, so random init misses that set with probability 1 (gradient descent never pushes you back into it either, so injectivity survives the entire training run)
@EmanueleRodola's group proves this specifically for the last hidden state at the final layer, but it goes further than that: any hidden state only depends on its own causal predecessors in that forward pass, so it knows all of them too (hidden state at position 5, layer 7 knows everything about the hiddens at positions 1-4 across layers 1 through 6)
https://t.co/bt4wL5QYSG