P

[Suwako — e/acc @suwakopro]

这篇论文有点意思啊，它证明注意力层和一个两层的MLP数学上等价，并且作为即时生成fast weight，然后宽度就是上下文长度，Q，K，V都分别对应着输入，第一层MLP的权重和第二层MLP的权重。softmax就是激活函数。

这一下就解释的通了，毕竟注意力机制这个矩阵看起来就不像是人脑里面能长出来的东西，看成MLP就合理多了。同时，这也说明了transformer里面其实就有现在持续学习里面的比较重要的fast weight路线。这么说的话，zero shot的能力其实也能解释的通了，解释为临时生成的MLP的能力，原来的一层attention一层MLP就是快慢权重的结合

更重要的是，现在的各种优化attention层的方法也可以看成是路由和压缩，类似linear attention就是MLP的压缩，而DSA就是路由，moe。

感觉未来也许我们能找到某个简单的方案，把transformer里面那些乱七八糟的东西都换成简单的MLP呢？这样也符合大道至简，符合bitter lesson。attention其实还是有人类的经验在里面的。
https://t.co/3FaySnWqwY

[P @P9472357898532]

@drizzle_lovely 그게 이유라면 볼버 싫어하는 사람한테 볼버하거나 아니면 좋아하는 사람이더라도 힘들때까지 죽어라 한다면 좀 나아지려나요?
근데 이러면 소프트인지 모르겠네요

[Math Files @Math_files]

In the 2024 Putnam Competition, none of the top 504 contestants earned full marks on Problem A6.

In February 2025, Grok 3 was reported to have solved it in eight minutes.

But its response checked a few cases and inferred a general formula without proving it.

Finding a pattern is not the same as proving a theorem.