This is an old hat. That's exactly what the neural fast weight programmers did in the 1990s (including the 1991 unnormalised linear Transformer). Overview with links to the original papers: https://t.co/pJ1raBlXmh
@neural_avb Could it be because Anthropic’s model was trained on LLM weights in line with their RSI work? The assumption that a “generalist” LLM can just work on “raw” weights out of the box seems naive, no?