@lossfunk Important problem and still long way to go to fully understand the phenomenon. Have been working on similar problem but from different angles, here are something might be good looking together: https://t.co/a27kYkdWdX
This interpretation is reasonable. In research of the self-loop behavior of small llms, we also notice that it may comes from in-context learning, which take previous prompt as patterns to follow rather than to refute, and worse, the more repeated times the stronger signal to follow. Some training tasks are just double-edged sword.
Introducing Nestor, a new generation of scalable architectures that generalize MoE transformers and explicitly promote sparsity in both attentions and experts, hence learn structured deep representations more efficiently. Work with @TianzheC and @druv_pai: https://t.co/NsjjPeX80k
@askalphaxiv v2 is now out🔥👉 https://t.co/ibmtmpmvQD
we animated the knowledge permeation and proved with a new algorithm that an aligment loss that distill late layer to midlayer boost generalization.
4/
Diagnosis: knowledge–circuit misalignment. Self-patching scans every layer pair at every checkpoint.
After memorization, the answer can already be read out OFF the diagonal: the fact is stored, but not in the middle layers where multi-hop composition happens.
👇
6/
LRSD turns this into training: on the SFT, actively align the head entity's representaiton at middle layer toward its own late layer.
Plain SFT goes flat soon after memorized; LRSD keeps climbing and almost double the generalization accuracy. Memorization not affected.
👇
3/
The problem: fine-tune an LLM on new facts and it recalls them soon, yet fails generalization questions that to use or reasoning with them, or lags far behind the memorization moment.
👇
Thanks @askalphaxiv for featuring us! 🙏
Now our v2 is out Today! Building on knowledge–circuit misalignment hypothesis, we derive LRSD, a practical SFT method that keeps boosting generalization by 2x after memorization saturates.
Project page: https://t.co/Cm4ii98dki
1/👇
“Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in LLM Finetuning”
LLMs can quickly memorize new facts during fine-tuning, but still fail when those facts need to be used in multi-hop reasoning.
This paper shows the issue is not missing knowledge, but knowledge–circuit misalignment. The fact exists inside the model, just not in the layers where reasoning happens.
Their self-patching method moves the right hidden representation into the right layer and recovers much of the lost generalization, with a simple heuristic recovering 58-75% of oracle headroom.
In fact, the first test-time compute I read is the test-time training (TTT). It was coined "dynamic evaluation" by Mikolov and used by Alex Graves / Ilya in old RNN/LSTM days. SoTA compression algorithms like PAQ-8 also had weight updates.