Clearly the models are using the dot tokens to do extra hidden computation (looking up facts, doing math)
We investigated how + what the models are doing and showed the computation is readable from the hidden states over the "meaningless" filler tokens: https://t.co/lpF5Yue9Zb
If you talk to any old priest who's actually been in the game forty years they're like yes the devil is real and he prefers intelligent people, don't pray when you're angry it opens the wrong door, some saints were terrifying in person and not in a good way, certain prayers work at certain times of night and we don't know why, and never let a possessed person know you're afraid
the book is: visual group theory, by nathan carter
read if you wanna see the soul of group theory
very minimal prereqs. being inquisitive is enough
10/10 fully recommended