Yunshui Li

🌱Bytedance Seed Researcher | Pre at 🥝 Qwen

[Lucas Beyer (bl16) @giffmana]

Love this type of work, nice one:

[Yunshui Li @cloud2water]

I’m genuinely curious about pretraining research at frontier AI labs in North America.

Assuming large-scale training runs are highly stable and require minimal manual intervention, what research directions do pretraining researchers typically focus on while these long runs are ongoing?

[Yunshui Li @cloud2water]

Was this report written by AI? Are you sure you reviewed it carefully? How could you make such a citation error?
https://t.co/AISU8OnJtP

[AI at Meta @AIatMeta]

Today we published a technical report on Muse Glimmer, our 30B open-weight model built for agentic tasks on consumer hardware.

It runs at practical speeds on a Mac or PC with a single consumer GPU at 4-bit precision, and leads comparably-sized open models on tool use and multi-step planning.

Read about the model architecture, distillation recipe, quantization and more: https://t.co/H7n9qeDhxn

[Yunshui Li @cloud2water]

@JamesZhang0365 @typelessdotcom 豆包输入法挺好的

[Yunshui Li @cloud2water]

@JamesZhang0365 Great job!

[Yunshui Li @cloud2water]

@JustinLin610 Best wishes, my bro

[Yunshui Li @cloud2water]

@box3490 Once the international version is released, I will get back to you as soon as possible

[Yunshui Li @cloud2water]

☘️ Seed 2.0 is finally released.

Over the past year, I think most of our progress came from advances in pretraining — continuous scaling led to continuous gains in intelligence.
Big thanks to our team for the journey together.

Blog: https://t.co/qRv5AQVedb

[Yunshui Li @cloud2water]

@SkyBlueHarbor https://t.co/cGyIczSsDK

[Yunshui Li @cloud2water]

@ysu_ChatData I think data is the foundation, but architecture and implementation details are what ultimately make scaling work. The gains from scaling are across the board. There are no plans to open-source it for now.

[Yunshui Li @cloud2water]

Model card: https://t.co/aoETx7Yvrm

[Yunshui Li @cloud2water]

Heading to San Diego for #NeurIPS2025! ☀️🌊

I’ll be presenting our work on Model Merging in Pre-training of LLMs.

📌 Poster #3411 (Exhibit Hall C,D,E)
⏰ Wed, Dec 3, 11:00 - 14:00 PST

I'm around all week (Dec 2-7). Feel free to DM me or stop by the poster. 👇

[Lucas Beyer (bl16) @giffmana]

They study model merging (EMA, soups), which is well-understood for fine-tunings of a base model, but they investigate *pre-training* of LLMs.

My summary thread.
(YoU wOn'T bEliEvE the surprise in post #5 which makes me like this group a lot more!)

[Yunshui Li @cloud2water]

Although trained with only 12T tokens, Seed-OSS achieves excellent performance on several popular open benchmarks. We release this series of models to the open-source community under the Apache-2.0 license.

[Yunshui Li @cloud2water]

🔥Seed-OSS is a series of open-source large language models developed by ByteDance's Seed Team, designed for powerful long-context, reasoning, agent and general capabilities, and versatile developer-friendly features.
https://t.co/YCW5AGeLVG

[Lucas Beyer (bl16) @giffmana]

This paper is pretty cool; through careful tuning, they show:
- you can train LLMs with batch-size as small as 1, just need smaller lr.
- even plain SGD works at small batch.
- Fancy optims mainly help at larger batch. (This reconciles discrepancy with past ResNet research.)
- At small batch, optim hparams are very insensitive!

I find this cool for two reasons:

1) When we did ScalingViT I also surprisingly found (but never published) that pure SGD works much better than expected. However, a small gap always remained, so we dropped it in favour of (our variant of) AdaFactor. The results here confirm this.

2) This is really good news for fine-tuning on small data with few GPUs. Drop the LoRA and do full fine-tuning with tiny batch-size and plain SGD!

A word of caution, because:

A) This is mostly done at tiny scale (30M params), to allow running many experiments. It is unclear how true the results remain at larger scale, although they do show a 1.3B result, it's usually after larger than 7B that things start to get more difficult.

B) This was all with transformers with QK-Norm, which has very stabilizing effect, I'd be curious if it holds without, but I give it a chance that it might.

C) For large-scale training, running on many (>10k) chips, large batch size is a necessity, not a choice. And they do show that at it's at large batch (not even that large: 4k) fancy optimizers matter significantly.

[Yuyu Zhang @_yuyuzhang]

🚀 Thrilled to introduce Seed-Coder, our new open-source family of 🔥powerful, 🔍transparent, ⚡parameter-efficient code models at 8B scale!

💥 Small model, big results!
🥇 BigCodeBench, FullStack Bench, MHPP: impressive results among lightweight open-source models, including Qwen3-8B, Qwen2.5-Coder-7B, DeepSeek-Coder-V2-Lite, and OpenCoder-8B.
🏆 SWE-bench Verified: 19.2% solve rate, proving that smaller LLMs can also be competent in complex software engineering scenarios.
🎖 IOI'24: 146.5 total score, surpassing leading open-source models such as QwQ-32B (144.0) and DeepSeek-R1 (137.0).
📊 Codeforces: 1553 ELO rating, close to o1-mini, showing the great potential of smaller LLMs on difficult reasoning tasks.

✨ Feel free to check it out, give it a spin, and share your thoughts! 💻

🌐 Homepage: https://t.co/jVL4FCCEMH
📄 Technical Report: https://t.co/Y9Eg5SzbV1
🔗 GitHub: https://t.co/2rKfmWYSDB
🤗 Models: https://t.co/6JgvnYclgt

[Yunshui Li @cloud2water]

🎓 And here’s the kicker: It aced the Gaokao New National Volume I Math exam with a 144/150—ranking #1 nationwide! 🚀💯
This is how AI empowers smarter, more adaptive solutions. What will you create with Doubao 1.6?

[Yunshui Li @cloud2water]

Exciting news from ByteDance! 🎉 Doubao Large Model 1.6 is officially live, introducing three powerful variants: Doubao-Seed-1.6-thinking, Doubao-Seed-1.6, and Doubao-Seed-1.6-flash.

🌟 Experience Link:
https://t.co/UA8JAaMFp5

#Doubao1Point6Launch

[Yunshui Li @cloud2water]

🌟 Doubao-Seed-1.6-thinking leads the pack with enhanced reasoning and multimodal support—perfect for tackling complex, creative tasks!
🧠🎥 ⚙️ Doubao-Seed-1.6 offers flexible thinking modes (on/off/auto) to adapt to your workflow seamlessly.