G

Former SOC analyst and DFIR. Currently Threat Hunting and Cloud ☁️ (☞ﾟヮﾟ)☞ My cats tweets

[Ahmad @TheAhmadOsman]

https://t.co/uRDNr16FHA

[Avid @Av1dlive]

Nvidia, Meta, OpenAI & Anthropic are secretly betting billions of dollars on this

This 55-minute Stanford lecture will teach you more about this secret architecture powering

the next ChatGPT than 2 months as an ML research intern at a frontier lab.

Bookmark it & give it time today. It'll be the most productive start for your week. Then read article below.

[Jason Zhu @GoSailGlobal]

Stanford CS336 上，Tatsu 讲了一节 LLM 架构课，把过去 3 年所有主流 LLM 拆开，看它们的共通模板

结论挺爆：90% 的架构选择已经收敛，你随便挑一个开源大模型，它跟其他模型在这些维度上几乎一模一样

讲师的原话
- 2024 年大家都在 cosplay Llama2
- 2025 年的主题是「怎么训得不崩」
- 2026 年的主题是「怎么扛住长上下文」

下面是 2026 年开源 LLM 的标准模板 你训自己的模型可以直接抄

【架构层 已经收敛的 7 件事】
1）Layer Norm 挪出残差流（pre-norm）
原版 Transformer 把 LN 放在残差里 几乎所有现代模型都挪到外面
原因：keep your residual stream clean 梯度反传更稳

2）RMS Norm 替代 LayerNorm
LayerNorm 的减均值 + 加 bias 那部分实际没怎么帮上忙
丢掉之后 flops 只省 0.17% 但运行时省到 25%
（瓶颈在数据搬运 计算反而次要）

3）所有 bias 项全删
跟 RMS Norm 一个道理 系统层省内存搬运

4）激活函数用 SwiGLU 或 GeGLU
gated linear unit 几乎所有现代模型都用
Llama 系 / Qwen / Mistral 用 SwiGLU
Google 系（Gemma / T5）用 GeGLU
区别极小 选哪个都行

5）位置编码用 RoPE
2024 年之后基本统一了
原理：把每对维度按位置旋转一个角度 让 inner product 只依赖相对位置

6）Transformer block 串联（不是并联）
GPT-J / Palm 试过并联 现在基本被放弃
串联的实现优化得太好了 并联省的那点系统开销不值得损失表达力

7）Layer norm 可以「撒」
哪儿不稳就在哪儿加 LN
attention 之前能加 之后能加 两边都加（double norm）也可以
现代模型很多这样做

【超参数 已经收敛的 5 个数】
1）feedforward 维度 / hidden 维度
- 非 GLU 模型：4 倍
- GLU 模型：8/3 ≈ 2.67 倍（因为 GLU 多一组矩阵 要保持总参数量）
- Llama 系：3.5 倍
- T5 1.0 试过 64 倍 后来 T5 1.1 改回标准 别学

2）head 数 × head 维度 ≈ hidden 维度
几乎所有模型都遵守 T5 是为数不多的例外

3）模型纵横比（hidden / 层数）≈ 100
太深 pipeline parallel 难做
太宽 表达力受限
100 这个数字是系统约束 + 表达力的平衡点

4）vocab size
单语模型：30K 左右（早期 GPT-2 那种）
多语 / 通用模型：100K-200K（GPT-4 / Llama 3 / Gemma 都在这个范围）
现代基本都是后者

5）weight decay
仍然普遍使用
但研究发现它在 LLM 里干的事其实是优化器干预 让你最终能收敛到更深的最优点
跟你想的「防过拟合」没什么关系
所以别因为「单 epoch 不会过拟合」就把它关掉

【稳定性 三个救命 trick】
训练大模型最怕中途 loss 突然飙升 然后 NaN 全军覆没
现代模型用三个 trick 防这件事

1）Z-loss
output softmax 的 normalizer 容易爆
加一个 (log Z)² 的正则项 让 Z 始终接近 1
DCLM / Olmo 都用

2）QK norm
attention 的 Q 和 K 在矩阵乘之前各加一个 LN
让 softmax 的输入永远是单位尺度
multimodal 圈先用起来 现在所有大模型都加

3）Logit soft cap（仅 Google 系）
attention logit 用 tanh 硬封顶
Gemma 2/3/4 都在用 但会损失一点点性能 慎用

【Attention 两个新趋势】
1）GQA（Grouped Query Attention）几乎统一
原版 multi-head 推理时 KV cache 会让算术强度崩到 1/h
GQA 共享 K 和 V 但保留多个 Q
表达力几乎不损失 推理成本砍掉 80%
现在所有要做生产部署的大模型 没有不用 GQA 的

2）局部 + 全局 attention 交替
处理长上下文的新方式
Cohere Command A 起头 现在 Llama 4 / Gemma 4 / Olmo 3 全在用
比如每 4 层有 1 层 full attention 其他 3 层是 sliding window 只看附近的 token
比纯 SSM 更稳 比纯 full attention 便宜得多
（Qwen 3.5 做了变体 把 sliding window 那 3 层换成 SSM）

收尾一句
如果你正在训自己的 LLM，上面这一套就是 2026 年的「默认配置」 不需要重新发明，直接抄

如果你只是想看懂 GitHub 上那些 modeling_xxx.py
这一份足够你不再被术语吓住

[Atal @ZabihullahAtal]

Stanford just released a 1.5-hour lecture on “LLM Architecture.”

This is the exact thing systems engineers at Anthropic and OpenAI require to understand at a deep level.

Give it some time.

This might be the highest-ROI learning you do this month.

[Dhaval Makwana @heyDhavall]

Instead of watching Netflix tonight, watch this 2-hour Stanford lecture.

You’ll learn more about how ChatGPT, Claude, and other LLMs are built than most people at top AI companies learn in years.

[Peter Holderrieth @peholderrieth]

🚀MIT Flow Matching and Diffusion Lecture 2026 Released (https://t.co/bKgs2wghvY)!

We just released our new MIT 2026 course on flow matching and diffusion models! We teach the full stack of modern AI image, video, protein generators - theory and practice. We include:
📺 Videos: Step-by-step derivations.
📝 Notes: Mathematically self-contained lecture notes
💻 Coding: Hands-on exercises for every component

We fully improved last years’ iteration and added new topics: latent spaces, diffusion transformers, building language models with discrete diffusion models.

Everything is available here: https://t.co/bKgs2wghvY

A huge thanks to Tommi Jaakkola for his support in making this class possible and Ashay Athalye (MIT SOUL) for the incredible production! Was fun to do this with @RShprints!

#MachineLearning #GenerativeAI #MIT #DiffusionModels #AI

[Ahmad @TheAhmadOsman]

There are maybe ~20-25 papers that matter.

Implement those and you’ve captured ~90% of the alpha behind modern LLMs.

Everything else is garnish.

You want that list? Look no more...

The Top 26 Essential Papers (+5 Bonus Resources)
for Mastering LLMs and Transformers

This list bridges the Transformer foundations
with the reasoning, MoE, and agentic shift

Recommended Reading Order
1. Attention Is All You Need (Vaswani et al., 2017)
> The original Transformer paper. Covers self-attention,
> multi-head attention, and the encoder-decoder structure
> (even though most modern LLMs are decoder-only.)

2. The Illustrated Transformer (Jay Alammar, 2018)
> Great intuition builder for understanding
> attention and tensor flow before diving into implementations

3. BERT: Pre-training of Deep Bidirectional Transformers (Devlin et al., 2018)
> Encoder-side fundamentals, masked language modeling,
> and representation learning that still shape modern architectures

4. Language Models are Few-Shot Learners (GPT-3) (Brown et al., 2020)
> Established in-context learning as a real
> capability and shifted how prompting is understood

5. Scaling Laws for Neural Language Models (Kaplan et al., 2020)
> First clean empirical scaling framework for parameters, data, and compute
> Read alongside Chinchilla to understand why most models were undertrained

6. Training Compute-Optimal Large Language Models (Chinchilla) (Hoffmann et al., 2022)
> Demonstrated that token count matters more than
> parameter count for a fixed compute budget

7. LLaMA: Open and Efficient Foundation Language Models (Touvron et al., 2023)
> The paper that triggered the open-weight era
> Introduced architectural defaults like RMSNorm, SwiGLU
> and RoPE as standard practice

8. RoFormer: Rotary Position Embedding (Su et al., 2021)
> Positional encoding that became the modern default for long-context LLMs

9. FlashAttention (Dao et al., 2022)
> Memory-efficient attention that enabled long context windows
> and high-throughput inference by optimizing GPU memory access.

10. Retrieval-Augmented Generation (RAG) (Lewis et al., 2020)
> Combines parametric models with external knowledge sources
> Foundational for grounded and enterprise systems

11. Training Language Models to Follow Instructions with Human Feedback (InstructGPT) (Ouyang et al., 2022)
> The modern post-training and alignment blueprint
> that instruction-tuned models follow

12. Direct Preference Optimization (DPO) (Rafailov et al., 2023)
> A simpler and more stable alternative to PPO-based RLHF
> Preference alignment via the loss function

13. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022)
> Demonstrated that reasoning can be elicited through prompting
> alone and laid the groundwork for later reasoning-focused training

14. ReAct: Reasoning and Acting (Yao et al., 2022 / ICLR 2023)
> The foundation of agentic systems
> Combines reasoning traces with tool use and environment interaction

15. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (Guo et al., 2025)
> The R1 paper. Proved that large-scale reinforcement learning without
> supervised data can induce self-verification and structured reasoning behavior

16. Qwen3 Technical Report (Yang et al., 2025)
> A modern architecture lightweight overview
> Introduced unified MoE with Thinking Mode and Non-Thinking
> Mode to dynamically trade off cost and reasoning depth

17. Outrageously Large Neural Networks: Sparsely-Gated Mixture of Experts (Shazeer et al., 2017)
> The modern MoE ignition point
> Conditional computation at scale

18. Switch Transformers (Fedus et al., 2021)
> Simplified MoE routing using single-expert activation
> Key to stabilizing trillion-parameter training

19. Mixtral of Experts (Mistral AI, 2024)
> Open-weight MoE that proved sparse models can match dense quality
> while running at small-model inference cost

20. Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints (Komatsuzaki et al., 2022 / ICLR 2023)
> Practical technique for converting dense checkpoints into MoE models
> Critical for compute reuse and iterative scaling

21. The Platonic Representation Hypothesis (Huh et al., 2024)
> Evidence that scaled models converge toward shared
> internal representations across modalities

22. Textbooks Are All You Need (Gunasekar et al., 2023)
> Demonstrated that high-quality synthetic data allows
> small models to outperform much larger ones

23. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (Templeton et al., 2024)
> The biggest leap in mechanistic interpretability
> Decomposes neural networks into millions of interpretable features

24. PaLM: Scaling Language Modeling with Pathways (Chowdhery et al., 2022)
> A masterclass in large-scale training
> orchestration across thousands of accelerators

25. GLaM: Generalist Language Model (Du et al., 2022)
> Validated MoE scaling economics with massive
> total parameters but small active parameter counts

26. The Smol Training Playbook (Hugging Face, 2025)
> Practical end-to-end handbook for efficiently training language models

Bonus Material
> T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2019)
> Toolformer (Schick et al., 2023)
> GShard (Lepikhin et al., 2020)
> Adaptive Mixtures of Local Experts (Jacobs et al., 1991)
> Hierarchical Mixtures of Experts (Jordan and Jacobs, 1994)

If you deeply understand these fundamentals; Transformer core, scaling laws, FlashAttention, instruction tuning, R1-style reasoning, and MoE upcycling, you already understand LLMs better than most

Time to lock-in, good luck!

[Arman Hezarkhani @ArmanHezarkhani]

https://t.co/Mt1PRTZCdT

[DAN KOE @thedankoe]

https://t.co/7l7Jef99QZ

[Lior Alexander @LiorOnAI]

Ilya Sutskever (OpenAI cofounder) top 30 must-read research papers.

"If you really learn all of these, you’ll know 90% of what matters today"

[Boris Cherny @bcherny]

I'm Boris and I created Claude Code. Lots of people have asked how I use Claude Code, so I wanted to show off my setup a bit.

My setup might be surprisingly vanilla! Claude Code works great out of the box, so I personally don't customize it much. There is no one correct way to use Claude Code: we intentionally build it in a way that you can use it, customize it, and hack it however you like. Each person on the Claude Code team uses it very differently.

So, here goes.

[Shruti @heyshrutimishra]

This paper didn’t go viral but it should have.

A tiny AI model called HRM just beat Claude 3.5 and Gemini.

It doesn’t even use tokens.

They said it was just a research preview.

But it might be the first real shot at AGI.

Here’s what really happened and why OpenAI should be worried: 🧵

[Brian Maierhofer @brianmaierhofer]

If your nervous system has been stuck in survival mode, you need a reset.

After years of stress, trauma, and modern life, it's forgotten how to truly relax.

Here are 16 specific techniques to reset it completely: 🧵

1) Neurogenic Tremoring (TRE)

[Poonam Soni @CodeByPoonam]

OpenAI, Google, and Anthropic just dropped the guides on:

- Prompt Engineering
- Building effective Agents
- AI in Business
- 601 AI use cases

and so much more...

9 best guides you don’t want to miss:

[Ivan Velichko @iximiuz]

This week, I silently released a CLI for iximiuz Labs 🚀

You can now start Linux, Kubernetes, Docker, and other types of DevOps playgrounds with a simple:

labctl playground start <name> --ssh

Haven't tested it much, so please report your findings 👇

https://t.co/SGnFZH8sOn

[MrBeast @MrBeast]

I’m gonna give 10 random people that repost this and follow me $25,000 for fun (the $250,000 my X video made)

I’ll pick the winners in 72 hours

[Michael Levan👨🏻‍💻☕️ @TheNJDevOpsGuy]

- Managing Kubernetes and deploying clusters WITH Kubernetes
- Using Kubernetes as a Platform Engineering IDP

and A LOT more.

Check out the playlist at the link below 👇

https://t.co/LBWHN9RbZn

[Ivan Velichko @iximiuz]

All right, it's deployed! Typically, it's 3 a.m. over here – indie dev at its finest 🙈

After a year of posting videos of my exploratory Kubernetes UI, I'm thrilled to finally share a link.

Go give it a try! 🚀

https://t.co/ieESYZEcRL

[Ronen Shustin @ronenshh]

Today we share our Alibaba Cloud research for the first time, where we gained unauthorized access to other customers' databases in two different services 🚨

This complex research involved RCE, PE, Container escape, K8s lateral movement, and supply chain attack. Check it out 🧵