Jerry Yin

B.A. @UMNComputerSci | Contributor of LLM Yi & SenseNova5o｜ex-Intern @ @ THUNLP @ SenseTime | MLSys & LLM | CUDA & Triton

[Jerry Yin @JerryYin777]

ChatGPT Work has quickly become one of my favorite productivity tools. It helps turn scattered ideas into clear plans, speeds up research and writing, and makes complex tasks feel much easier to manage. It’s like having a smart, reliable teammate ready whenever you need one.

[Jerry Yin @JerryYin777]

@JustinLin610 OMG

[Jerry Yin @JerryYin777]

@diegocabezas01 @n1d_all Hi Diego, can you send me a code, I'll send it below, thanks!

[Songlin Yang @SonglinYang4]

Flash Linear Attention (https://t.co/k2UDXQAaKo) will no longer maintain support for the RWKV series (existing code will remain available). Here’s why:

[Jerry Yin @JerryYin777]

I’m incredibly excited that I have officially earned my Bachelor of Arts in
Computer Science with Distinction at UMN!

I’m grateful for my experience. It has been a wonderful journey, and I think I will remember everything about this place (except for the extremely cold lol).

[Daniel Han @danielhanchen]

Cool things from DeepSeek v3's paper:

1. Float8 uses E4M3 for forward & backward - no E5M2
2. Every 4th FP8 accumulate adds to master FP32 accum
3. Latent Attention stores C cache not KV cache
4. No MoE loss balancing - dynamic biases instead

More details:
1. FP8: First large open weights model to my knowledge to successfully do FP8 - Llama 3.1 was BF16 then post quantized to FP8.

But method different - instead of E4M3 for forward and E5M2 for backward, used ONLY E4M3 (exponent=4, mantissa=3).

Scaling is also needed to extend the range of values - 1x128 scaling for activations and 128x128 scale tile for weights.

During used per tensor scaling, and other people use per row scaling.

2. FP8 accumulation errors: DeepSeek paper says accumulating FP8 mults naively loses precision by 2% or more - so every 4th matrix multiply, they add it back into a master FP32 accumulator.

3. Latent Attention: Super smart idea of forming the K and V matrices via a down and up projection! This means instead of storing K and V in the KV cache, one can store a small slither of C instead!

C = X * D
Q = X * Wq
K = C * Uk
V = C * Uv

During decoding / inference, in normal classic attention, we concatenate a new row of k and v for each new token to K and V, and we only need to do the softmax on the last row.

Also no need to form softmax(QK^T/sqrt(d))V again, since MLP, RMSNorm etc are all row wise, so the next layer's KV cache is enough.

During inference, the up projection is merged into Wq:
QK^T = X * Wq * (C * Uk)^T
= X * Wq * (X * D * Uk)^T
= X * Wq * Uk^T * D^T * X^T
= (X * (Wq * Uk^T)) * (D^T * X^T)
And so we can pass these 2 matrices to Flash Attention!

4. No MoE loss balancing: Instead of adding a loss balancer, DeepSeek instead provides tuneable biases per expert - these biases are added to the routing calculation, and if one expert has too much load, then the bias will be dynamically adjusted on the fly to reduce it's load.

There is also sequence length loss balancing - this is added to the loss.

5. Other cool things:
a) First 3 layers use normal FFN, not MoE (still MLA)
b) Uses DualPipe for 8 GPUs in a node to overlap communication and computation
c) 14.8 trillion tokens - also uses synthetic data generation from DeepSeek's o1 type model (r1)
d) Uses YaRN for long context (128K). s = 40, alpha = 1, beta = 32 - scaling factor = 0.1*log(s) + 1 ~= 1.368888

DeepSeek v3 paper: https://t.co/q6fJ8Eu2og

Also happy holidays!!

[1a3orn @1a3orn]

@teortaxesTex I can't find anyone executing on multi-token prediction after the original Meta paper, this seems to be the first large scale validation of the idea.

Just really clean.

https://t.co/54oQR6uiBN

[Tri Dao @tri_dao]

Hadamard transform was the very first kernel I wrote when learning CUDA many years ago. It convinced me that there's sth fun and fundamental about ML and systems. Now you can even use tensor cores for these fast transforms

[HPCwire @HPCwire]

NVIDIA: Blackwell Delivers Next-Level MLPerf Training Performance
https://t.co/V5bXD1Ttu7 #NVIDIA #HPC #TCIwire #Blackwell #MLPerf

[Paul Gauthier @paulgauthier]

The new Qwen 2.5 Coder models did very well on aider's code editing benchmark. The 32B Instruct model scored in between GPT-4o and 3.5 Haiku.

84% 3.5 Sonnet
75% 3.5 Haiku
74% Qwen2.5 Coder 32B
71% GPT-4o
69% Qwen2.5 Coder 14B
58% Qwen2.5 Coder 7B

https://t.co/mBVaUPGHPl

[James Zou @james_y_zou]

How well do #LLM fine-tuning APIs work? Not well.

We created #FineTuneBench to study if we can update #ChatGPT, #Gemini's knowledge or teach it new facts via finetuning APIs. Answer is mostly No.

Finetune 4o-mini was the most effective; Gemini less https://t.co/PKladFsctv Users want finetuning-as-a-service but current commercial APIs are limited wrt knowledge infusion. While OpenAI APIs can memorize, generalization is poor. We hope #FineTuneBench can be a useful metric and leaderboard to improve these services https://t.co/p54ejqzMcG

Great work by @kevinywu @ericwu93 👏

[zhyncs @zhyncs42]

Excited to have hosted the SGLang @lmsysorg talk yesterday with @GPU_MODE 's @marksaroufim and @hsu_byron. Thanks to FlashInfer @ye_combinator and LMDeploy @intern_lm teams for their support!
https://t.co/EYzXR2lHqj

[Jerry Yin @JerryYin777]

@realHongyu_Wang Cool

[Hongyu Wang🥕 @realHongyu_Wang]

👉🏻Thrilled to introduce BitNet a4.8, enabling 4-bit activations for 1.58-bit LLMs!🚀🚀

Paper: https://t.co/jHGRKCub96
HF page: https://t.co/Zqb2hLZ5K9

🔥🔥2B BitNet a4.8 trained with 2T tokens achieves 50.30% acc on MMLU, almost no degradation to BitNet b1.58.

[Jerry Yin @JerryYin777]

@kaloronaHuang 你醒了 现在是2016年 你怎么嘴里天天念叨着什么美国 什么CS PhD 什么大包 快起来出去玩儿或者打两把游戏吧

[Anthropic @AnthropicAI]

Introducing an upgraded Claude 3.5 Sonnet, and a new model, Claude 3.5 Haiku. We’re also introducing a new capability in beta: computer use.

Developers can now direct Claude to use computers the way people do—by looking at a screen, moving a cursor, clicking, and typing text.

[Santiago @svpino]

Large Language Models don't reason.

Thank you,
Apple.

[Jerry Yin @JerryYin777]

@kaloronaHuang 办签证/办OPT/赶3+Days以上的Due同感😊

[Shishir Patil @shishirpatil_]

💥 LLAMA Models: 1B IS THE NEW 8B 💥

📢 Thrilled to open-source LLAMA-1B and LLAMA-3B models today. Trained on up to 9T tokens, we break many new benchmarks with the new-family of LLAMA models. Jumping right from my PhD at Berkeley, to train these models at @AIatMeta has been an exhilarating transition :)

Curious about how we train them? What are the challenges of extending 1B models to longer context lengths? Do scaling laws from 405B and 8B apply to 1B?What does "post-training saturation" look like? And of-course how well do these models perform on tooling?

🧑‍🔬 Key Details ⌨️

Pre-training ✂️💡
We prune the models from their 8B siblings and use logits from the 8B and 70B models as token-level targets (token-level distillation). We then use knowledge distillation to recover performance.

🍪Tokens: The 1B (1.23B) and 3B (3.21B) models, are both trained on up to 9T tokens for 370K and 460K training hours respectively.

CPT and Long-Context: 🪘 ⚖️
We extend the 8K pre-trained model to 128K context. This is challenging as at such smaller scales, as large-context benefits (SCROLLS, InfiniteBench, NIAH) come at the cost of short-context performance (GSM-8K, GPQA, MMLU). Solution: Continually Pre-Train a very short warm-up, but high-LR model to get long-context gains, and then Continually Pre-train a long warm-up, low-LR model for short-context gains, and soup the two models intelligently - metrics and good vibes 😎
Easter egg: What happens at 524K? If we set the rope scale factor to 64, can we hit 1M context 🦣 at 1B 👶

Post-training 🧩🎯
Training smaller models is a very different beast given they demonstrate angular profiles, especially at 1B scale. Although the training itself could be made stable, there are real trade-offs to be made! A gain in instruction following (IFEval) might come at the expense of coding (MBPP, Human-eval)! This is a useful exercise to understand what model saturation would look like! Perhaps, we'll soon have to start making such choices for larger models as well!

Tooling 🛠️🚀
Naturally, you’d expect my models to excel in tooling! 😜 LLAMA-1B and LLAMA-3B set new benchmarks in function-calling, reaching 8B levels!

Training a foundation model alongside amazing collaborators has been an incredible journey. I hope you’ll enjoy working with these tiny beasts! 💥

[Mira Murati @miramurati]

I shared the following note with the OpenAI team today.