Pingbang Hu 🇹🇼

I work on, with, and for data. Ph.D. candidate @UofIllinois. Fellows @AnthropicAI. Interns @ SIG @amazon @jouhouken. Alumni @Umich @SJTU1896.

[Pingbang Hu 🇹🇼 @PingbangHu]

New Preprint Alert ⏰

Propose Dr. Post-training 🩺 a Data Regularization framework, making your data more effective with ZERO overheads

Experiments demonstrate faster training convergence across SFT, RLHF, RLVR over SOTA data selection, opening up new data optimization designs!

[Pingbang Hu 🇹🇼 @PingbangHu]

Will spend the summer at Susquehanna (SIG) as an ML/QR intern. Starting tmr.

I will not be a finance bro I promise.

[Pingbang Hu 🇹🇼 @PingbangHu]

@HelloVyom sorry but I'm pretty sure deepseek (or High-Flyer) is not the biggest "quant" competitors of USA.

[Pingbang Hu 🇹🇼 @PingbangHu]

@kaiqu_liang previous fellow here. Enjoy!

[Anthropic @AnthropicAI]

We've raised $65 billion in Series H funding at a $965 billion post-money valuation, led by @AltimeterCap, Dragoneer, @Greenoaks, and @sequoia.

This investment will help us advance our research and expand our capacity to meet growing demand for Claude.

[Pingbang Hu 🇹🇼 @PingbangHu]

New Preprint Alert ⏰

Propose Dr. Post-training 🩺 a Data Regularization framework, making your data more effective with ZERO overheads

Experiments demonstrate faster training convergence across SFT, RLHF, RLVR over SOTA data selection, opening up new data optimization designs!

[Pingbang Hu 🇹🇼 @PingbangHu]

@joemelko for real

[Pingbang Hu 🇹🇼 @PingbangHu]

while ago @joemelko told me that the post-training technique I'm working on (https://t.co/SJOcHdEWR4) will also work in pretraining, if not then it's skill issue.

now given this promising signal I'm ready. only problem is where's the gpu credit 😭

[机器之心 JIQIZHIXIN @jiqizhixin]

There is now a smarter way to pick data for training LLMs!

Enter OPUS!

This is an ICML Oral paper from SJTU, Alibaba, UW–Madison, UIUC, and Mila - Quebec AI Institute.

The proposed method dynamically and intelligently selects the most impactful data for LLM pre-training in every single training iteration, bringing principled, continuous data optimization to the forefront.

This approach aims to significantly boost training efficiency and yield higher-quality LLMs, outperforming conventional static data selection methods across diverse language tasks.

OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

Paper: https://t.co/zgAzuwoTJf

Our report: https://t.co/tUCDBOHV5q

📬 #PapersAccepted by Jiqizhixin

[Pingbang Hu 🇹🇼 @PingbangHu]

@ShaoboWang6 @joemelko happy to

[Pingbang Hu 🇹🇼 @PingbangHu]

This is really my best work so far and am generally proud of sharing this with people who are interested.

Side note: I'm right now in Anthropic team matching process after the fellowship if someone know teams that are doing data please dm me!!!

[Pingbang Hu 🇹🇼 @PingbangHu]

BTW, we have a blog post as well!
Please check out in case the paper is too long 😗

Paper: https://t.co/Nqo4aFcEnf
Blog: https://t.co/t1OGqzZhuz

[Pingbang Hu 🇹🇼 @PingbangHu]

Day 140

I left a little fan for the next tenant, enjoy.

[Pingbang Hu 🇹🇼 @PingbangHu]

Day 1 in SF

At least I got a bed

[Andrej Karpathy @karpathy]

Personal update: I've joined Anthropic. I think the next few years at the frontier of LLMs will be especially formative. I am very excited to join the team here and get back to R&D. I remain deeply passionate about education and plan to resume my work on it in time.

[Pingbang Hu 🇹🇼 @PingbangHu]

@jparkjmc FOMO

[Pingbang Hu 🇹🇼 @PingbangHu]

@HighFreqAsuka fair 🤣

[Pingbang Hu 🇹🇼 @PingbangHu]

@joemelko @HighFreqAsuka im listening

[Joe Melkonian @joemelko]

fun little artifact, i worked on something similar to freon last year and started writing an (unedited) post that is hidden on my blog: https://t.co/4flVlmPEnC

very naive implementation of steepest descent under various p using full svd: https://t.co/civIcS5XP4

[Pingbang Hu 🇹🇼 @PingbangHu]

BTW, we have a blog post as well!
Please check out in case the paper is too long 😗

Paper: https://t.co/Nqo4aFcEnf
Blog: https://t.co/t1OGqzZhuz

[Pingbang Hu 🇹🇼 @PingbangHu]

New Preprint Alert ⏰

Propose Dr. Post-training 🩺 a Data Regularization framework, making your data more effective with ZERO overheads

Experiments demonstrate faster training convergence across SFT, RLHF, RLVR over SOTA data selection, opening up new data optimization designs!

[Pingbang Hu 🇹🇼 @PingbangHu]

@iamashtonchew Glad you like it!!

[Pingbang Hu 🇹🇼 @PingbangHu]

New Preprint Alert ⏰

Propose Dr. Post-training 🩺 a Data Regularization framework, making your data more effective with ZERO overheads

Experiments demonstrate faster training convergence across SFT, RLHF, RLVR over SOTA data selection, opening up new data optimization designs!