Jiaxuan Zou

Pretrain and Scaling Research @ByteDance Seed

[Jiaxuan Zou @SmartPig_Joe]

Life update: I will join the Pretrain team of ByteDance Seed (based in Beijing) in early July to keep doing research on LLM pre-training. I'm looking forward to collaborating with elite professionals and contributing to social value! 😀🫡

Welcome to have a coffee chat!

[Percy Liang @percyliang]

Marin 535B-A23B is ~7% done training, and so far things look on track. Next Tuesday (Sept 1 @ 10 PT), we will have a Zoom panel/discussion where the Marin team will talk about the design decisions that went into this run, the tradeoffs made, and our learnings. If you're interested, join the Discord for details (https://t.co/xwJSxd2IxH).

[Jiaxuan Zou @SmartPig_Joe]

@zhanpeng_zhou This is a very interesting topic.

[Jiaxuan Zou @SmartPig_Joe]

2/2
It explains that hyperball is actually just a special learning rate scheduler. It also explains why this peak-decay shaped learning rate schedule often appears—bias-variance trade-off.

[Jiaxuan Zou @SmartPig_Joe]

1/2
https://t.co/IY2w6D6Uf4 Lei Wu's new paper, and I would also like to take this opportunity to promote my recently written blog, https://t.co/92vpdxoMsD
This blog post compiles recent research on ELR and hyperball.

[Jiaxuan Zou @SmartPig_Joe]

@_selebou @rosinality true

[Rosinality @rosinality]

https://t.co/B7qdNg62Gw

Effective learning rate, the ratio of learning rate and weight norm, governs training dynamics. This could allow transferring the settings across norm control methods (by matching effective learning rate).

[Jiaxuan Zou @SmartPig_Joe]

Scaling is a methodology, even a scientific outlook. It denotes a scientific way of turning energy into intelligence, and is not confined to LLMs. This work uses Hyperball (Kaiyue et al.) as the optimizer, which has shown more stable scaling behavior across multiple scenarios.

[Jiaxuan Zou @SmartPig_Joe]

Unlike muon, muon requires performing momentum first, followed by NS iteration. Momentum plays a denoising role at the spectrum level, enabling better convergence of NS iteration.

[Liu Ziyin @LiuZiyin10]

What if momentum accumulation in Adam is "wrong"? In 2020, we proposed LaProp. It performs momentum accumulation after normalization, which makes training much more stable across all parameter ranges - maybe it could be useful today.

Read it here: https://t.co/fVmzXuGm3a

[Yuhan Ye @YyuhanYe]

Last week, Ma and Chen proved the first \omega(n^{-2}) lower bound for (non-anytime) GD with predetermined stepsizes. Soon after, Chung-En Tsai improved it to \Omega(n^{-sqrt{3}). With the help of ChatGPT-5.6 Sol, I further improved it to \Omega(n^{-1.6342}).

[Jiaxuan Zou @SmartPig_Joe]

@shirshakchavan SF, for sure

[Jiaxuan Zou @SmartPig_Joe]

@willcb I believe the common practice is to incrementally increase the seq_len, which is a form of curriculum learning to some extent.

[Jiaxuan Zou @SmartPig_Joe]

I now believe that long‑context capabilities stem more from pre-training corpora rather than architecture.

[Alexander Whedon @alex_whedon]

Introducing SubQ - a major breakthrough in LLM intelligence.

It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA),

And the first frontier model with a 12 million token context window which is:

- 52x faster than FlashAttention at 1MM tokens
- Less than 5% the cost of Opus

Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention).

Only a small fraction actually matter.

@subquadratic finds and focuses only on the ones that do.

That's nearly 1,000x less compute and a new way for LLMs to scale.

[Jiaxuan Zou @SmartPig_Joe]

@flyingpetal472 @liulicheng10 仿生的东西在 ai 里不一定 work，如无必要，勿增实体

[Jiaxuan Zou @SmartPig_Joe]

1/n
Watching a recent interview with a frontier-lab researcher left me with a list of questions — not for that one person, but for anyone working on frontier models in industry, myself included. The most important one is at the end. 🧵

[Jiaxuan Zou @SmartPig_Joe]

@coderinblack I cannot provide quantitative results or papers to back this up, but my industry‑level experience and my taste tell me this is the case.

[Jiaxuan Zou @SmartPig_Joe]

@flyingpetal472 是滴，各大厂都很重视long ct 数据，包括我们

[Jiaxuan Zou @SmartPig_Joe]

@zhanpeng_zhou truly valuable message

[Jiaxuan Zou @SmartPig_Joe]

@chemeris yes

[elie @eliebakouch]

marin is the most open model training initiative in the world

you can visualize the exact pre training mixture per domain, look at sampled documents, follow the live training loss, dig into configs, scaling laws

[Jiaxuan Zou @SmartPig_Joe]

@flyingpetal472 scaling也能算上半门科学

[Jiaxuan Zou @SmartPig_Joe]

Put differently: do frontier labs possess a "science of models" the outside world doesn't yet know — or do they possess the organizational capability to execute a large body of already-known methods reliably and to the extreme?🧐🧐🧐

[Jiaxuan Zou @SmartPig_Joe]

8/n — The most important question
How much of today's frontier labs' technical lead comes from scientific understanding that hasn't been made public, and how much comes merely from more compute, internal data, evaluation systems, and engineering infrastructure?