Yang Wang

Microsoft Research

[Yang Wang @YangWang92]

@mc_mozer Thank you for the confirmation and the update! I also believe that the backend benchmark won't be affected by the BOS issue. Thanks for such interesting work on Recirculation, it is fantastic to see such effective training-free methods.

[Yang Wang @YangWang92]

(1/6) A reproduction of Recirculation https://t.co/XZ2wH4j6dM on the Gemma 3 1B PPL test and can confirm the initial PPL drop (injecting source layer 11 to dst layer 4 works nicely).

[Yang Wang @YangWang92]

(6/6) Ablation on Gemma 3 show that including the BOS token negates Recirculation's performance gains. Further tests on injected vectors confirm that BOS injection best reproduces the effect.

[Yang Wang @YangWang92]

(5/6) Tried adding the BOS token back to reproduce Table 1, and the results are quite interesting! If we only add BOS to the first window, the PPL roughly matches the paper. But if we add it to every eval window, the PPL drops significantly.

[Yang Wang @YangWang92]

(4/6) I inserted a BOS token into each evaluation window. The baseline PPL scaling returned to normal across all model sizes. (1B → 4B → 12B，arxiv 12.935 → 8.562 → 7.262)

[Yang Wang @YangWang92]

(3/6) Looking at the attention heatmap, it's clear that the BOS token acts as a attention sink. I guess during evaluation, when long sequences were chunked into windows, perhaps only the first eval chunk had a BOS token.

[Yang Wang @YangWang92]

(2/6) However, I noticed a few puzzling details. In Table 1, the 12B model shows a higher PPL than the 1B/4B models, and the method's gain vanishes on the Lambada. Since the eval window is 1024, I checked Lambada dataset, and its average length is indeed < 1024.

[Yang Wang @YangWang92]

(6/6) Ablation on Gemma 3 show that including the BOS token negates Recirculation's performance gains. Further tests on injected vectors confirm that BOS injection best reproduces the effect.

[Yang Wang @YangWang92]

(5/6) Tried adding the BOS token back to reproduce Table 1, and the results are quite interesting! If we only add BOS to the first window, the PPL roughly matches the paper. But if we add it to every eval window, the PPL drops significantly.

[Yang Wang @YangWang92]

(5/6) Tried adding the BOS token back to reproduce Table 1, and the results are quite interesting! If we only add BOS to the first window, the PPL roughly matches the paper. But if we add it to every eval window, the PPL drops significantly.

[Yang Wang @YangWang92]

(4/6) I inserted a BOS token into each evaluation window. The baseline PPL scaling returned to normal across all model sizes. (1B → 4B → 12B，arxiv 12.935 → 8.562 → 7.262)

[Yang Wang @YangWang92]

(4/6) I inserted a BOS token into each evaluation window. The baseline PPL scaling returned to normal across all model sizes. (1B → 4B → 12B，arxiv 12.935 → 8.562 → 7.262)

[Yang Wang @YangWang92]

(3/6) Looking at the attention heatmap, it's clear that the BOS token acts as a attention sink. I guess during evaluation, when long sequences were chunked into windows, perhaps only the first eval chunk had a BOS token.

[Yang Wang @YangWang92]

(3/6) Looking at the attention heatmap, it's clear that the BOS token acts as a attention sink. I guess during evaluation, when long sequences were chunked into windows, perhaps only the first eval chunk had a BOS token.

[Yang Wang @YangWang92]

(2/6) However, I noticed a few puzzling details. In Table 1, the 12B model shows a higher PPL than the 1B/4B models, and the method's gain vanishes on the Lambada. Since the eval window is 1024, I checked Lambada dataset, and its average length is indeed < 1024.

[Yang Wang @YangWang92]

(2/6) However, I noticed a few puzzling details. In Table 1, the 12B model shows a higher PPL than the 1B/4B models, and the method's gain vanishes on the Lambada. Since the eval window is 1024, I checked Lambada dataset, and its average length is indeed < 1024.

[Yang Wang @YangWang92]

(1/6) A reproduction of Recirculation https://t.co/XZ2wH4j6dM on the Gemma 3 1B PPL test and can confirm the initial PPL drop (injecting source layer 11 to dst layer 4 works nicely).

[Yang Wang @YangWang92]

@huskydogewoof @xidulu @qubitium I'm reproducing the recirculation results too. Will update with my numbers soon. 😆

[Yang Wang @YangWang92]

@LalitMaganti Thanks for the corrections! It is a vibed project haha, mostly to make checking complex traces easier for myself since model training traces are absolutely massive. Thanks for building Perfetto!

[Yang Wang @YangWang92]

Vibed a small tool for viewing remote PyTorch traces (Perfetto) right from your local machine: https://t.co/tIxH8VhmWa. Say goodbye to dragging gigabytes of trace files back to your laptop. Parsing is handled entirely on the server side, which saves bandwidth and prevents OOM.

[Yang Wang @YangWang92]

It is astonishing that extended extrapolation can be achieved through such a simple and elegant method!

[Xidulu @xidulu]

1/ Sharing a new, interesting project I did during my internship at Microsoft AI Frontiers w/ @JohnCLangford

TL;DR: At decoding time, we feed **previous hidden state** into the input together with token embedding, and it boosts performance for free.

https://t.co/za3cZ4UwWV

[Yang Wang @YangWang92]

@gurtej__gill_ pretraining phase1/2, long ctx and annealing?

[Liyan Chen @liyanc98]

IHC at 17.5TB/s! :bufo-excited:

[Yang Wang @YangWang92]

@wuyang_zhou Our experiments are primarily based on this PR: https://t.co/i9mMgexjs1. We are still evaluating more configurations, thanks for sharing your work!

[Tian Xie (Unakar) @unakar666]

DeepSeek mHC: dropping the "m" improves performance.
Empirical finding:  identity hres outperforms the original design. Concurrent discoveries by multiple teams confirm this
https://t.co/MsqkS9W9Cs