Yingfa Chen

Intern at @OpenBMB. PhD at @Tsinghua_Uni. Working on #ML & #NLProc

[Albert Gu @_albertgu]

it's hilarious how many have taken the bitter lesson to mean just scaling up transformers. it's about scaling, but innovations in algorithmic primitives can lead to much more efficient scaling

[rohan anil @_arohan_]

Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models.

At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training.

I hope you all enjoy this write up!🪐🛰️

https://t.co/gzcwn27Pxy

[Nikita Pavlichenko @nv_pavlichenko]

Finally releasing the results of the arch ablations I was running with bux over the weekend https://t.co/S5VSc51uxN

[Yingfa Chen @yingfachen]

@loiccabannes Thanks for the attention, loved reading your paper back then!
We have internally tried some other interventions like using different LRs for different layers, but it did not yield further improvements. We are still actively working on this.

[Yingfa Chen @yingfachen]

Stronger linear attention can make hybrid LLMs weaker. Why?

This weird finding has been on my mind for a long time. We finally have a hypothesis that could explain it, and experiments that support it.

Here’s the story. 🧵
(1/5)

[Yingfa Chen @yingfachen]

@bezdarnost_brr By "stronger", we meant that when KDA gets stronger during training, it reduces the learning speed of other layers. We do have results with GDN, and we are doing experiments with Mamba now! Thanks for the suggestions!

[Yingfa Chen @yingfachen]

@ahatamiz1 Thanks for the suggestion! GDN-2 wasn't released when we ran the expeirments, but it's already in our schedule, will be trying it out as soon as we get the resources!

[Yingfa Chen @yingfachen]

@ahatamiz1 @teortaxesTex supporting 100M is easy, but it has to outperform, for instance, 1M context window + a good harness. wonder how you're gonna solve this with one person🤔

[Yingfa Chen @yingfachen]

@ahatamiz1 @teortaxesTex but we lack high-quality 100M data

[Yingfa Chen @yingfachen]

💡 Who learns first can shape what the whole model learns.

If you train hybrid LLMs, we’d love to hear whether you observe similar competition!

📄 Paper: https://t.co/hEyAFpbgd4
💻 Code: https://t.co/CK5HVXvK2B
(5/5)

[Yingfa Chen @yingfachen]

📈 Across six KDA hybrids (180M–1B), BLD improves RULER and recall at every size.

At 1B:
RULER: 32.75 → 36.04
Recall: 36.50 → 38.14
LongPPL: 7.69 → 6.50 ↓
Gated DeltaNet and GLA hybrids improve too.
(4/5)

[Junyang Lin @JustinLin610]

a life update: i started a new company called Pragmatik (p7k) Labs (语用科技) in shanghai, focusing on the research of next-generation agents across digital and physical worlds. thanks to Gaorong Ventures and HSG (红杉中国 & 高榕创投) for co-leading this round, and to Tencent (腾讯) and Shanghai Engine Fund (上海未来产业基金) for the support. @pragmatik_labs · https://t.co/01J3uSbxmX

[DAIR.AI @dair_ai]

https://t.co/bVYEuDFWY1

[Yingfa Chen @yingfachen]

@eric_alcaide @herrywen1 @nopainkiller @zhangyir22 Figure 6 shows the result where line breaks are replaced with star separator, in which case 2D-RoPE reduces to RoPE, so it should have similar performance to RoPE

[Haodong Wen @herrywen1]

(1/n)If you paste your data into an LLM and ask it to plot it in Python, it first has to copy every number into the code — so can you trust the result, or might it quietly miscopy a few?

Our new paper finds frontier LMs sometimes do get this wrong! Models that can solve hard reasoning problems still fail at a much simpler task: copying an input exactly, token by token.

Why? RoPE has an inductive bias that works against exact copying.
Better option? we propose 2D-RoPE, which views text as 2D instead of 1D.

[Nathan Barry @nathanrs]

I found out the other day that any compression tool can be contorted to do language modeling. Turns out gzip can generate text that somewhat *resembles* Shakespeare. Short write up linked below

[Yingfa Chen @yingfachen]

@sheriyuo @paperpaper886 比 opus 便宜太多了

[Jianyang Gao @gaoj0017]

The TurboQuant paper (ICLR 2026) contains serious issues in how it describes RaBitQ, including incorrect technical claims and misleading theory/experiment comparisons.

We flagged these issues to the authors before submission. They acknowledged them, but chose not to fix them. The paper was later accepted and widely promoted by Google, reaching tens of millions of views.

We’re speaking up now because once a misleading narrative spreads, it becomes much harder to correct. We’ve written a public comment on openreview (https://t.co/nDVjmNhATM).

We would greatly appreciate your attention and help in sharing it.