Albert

喜欢音乐

[Albert @yangAdam9]

@wwwgoubuli 思考，行动，观测本来就是个常识。这哥们完全是因为占坑早，GPT3.5还没出就发了，后面不得不引罢了。

[Albert @yangAdam9]

@NFT_Chen 你特么吓我一跳，这是参数merge? 这不过是在MOPD阶段加了一个专家模型。 不要在这里乱说。

[Albert @yangAdam9]

@_LuoFuli Why does it look so similar to DeepSeek V4.1 Flash?

[Fuli Luo @_LuoFuli]

MiMo-V2.6: The Hard Road to Scaling Up RL

MiMo-V2.6 is very likely one of the largest single RL runs, by compute, that any open-source model team has undertaken to date. In an era when compute is brutally scarce, we still chose to dedicate a team of several dozen people to one goal over an extended period: scaling up RL. That takes more than research conviction. It takes a vision for AGI, respect for the unknown, and the nerve to walk straight into the hardest problems.

The result is a model whose potential was built through mid-training and unlocked through heavy RL. Today, it is the number one open-source model. I strongly recommend reading the technical report. I believe it will become one of those papers that Agent RL practitioners keep reopening and discovering something new in each time. In my view, the research innovations and engineering challenges behind it surpass those of DeepSeek R1, which I was partly involved in.

Some will ask: why MixRL instead of MOPD? First, they are not competing choices. We ran MixRL on verifiable tasks of moderate difficulty, including code and related agentic tasks, and found that the resulting models generalize remarkably well. Second, tasks that are difficult to verify, extremely long-horizon, or simply too challenging to include in a joint RL run are trained separately. Including them would substantially reduce rollout efficiency or introduce significant rollout staleness. We then merge the resulting capabilities through MOPD. Games, 3D tasks, and tasks with subjective evaluation signals all fall into this category.

There is also a third, slightly cheeky answer. Our team is flat enough and free enough of organizational silos that MixRL simply is not difficult for us. More importantly, everyone enjoys working this way. People from different domains come together every day, driven by the pursuit of AGI and intelligence that can continuously improve itself, to confront and resolve the RL bottlenecks in each field. I will always remember the RL daily update meetings from this period. They were intense and dense, with intelligence emerging in real time.

To help the open-source community focus on solving real Agentic RL problems, we have released a Qwen model distilled from MiMo RL trajectories as a stronger starting point for RL, along with 7K diverse environments and a complete RL training framework. We hope these resources will help move Agentic RL research forward.

MiMo-V2.6 is only the beginning. In an era when intelligence is easy to replicate, we still choose the hard road toward self-improvement and AGI. Much of what lies ahead remains unknown. But we are willing to keep investing the time, compute, and passion required to take on one hard problem after another and work each of them all the way through, until intelligence crosses into a new regime.

[Dustin Tran @dustinvtran]

MiMo-V2.6: If anyone wants to experience what it's like to captain an RL run (train a checkpoint that actually gets launched), this is it. Very similar to my experience at xAI and Gemini: no run is ever launch-and-forget. It is how many infra and recipe issues you can discover and patch while the run is going. Hard failures are easiest, if you don't think there's an issue, you don't have enough metrics or aren't spot-checking enough traces.

[Fuli Luo @_LuoFuli]

Nearly half a year of silence. We spent it studying one problem: how far RL can scale.

MiMo-V2.6 is in the middle of its RL run right now. Three things we scaled: compute (~2B tokens per step, 1568 prompts × 16 rollouts, fully async), environments and harnesses (multi-task agentic RL, mixed across multiple harnesses in one run), and grader compute (agentic in-group credit assignment, with test-case and rubric-based rewards). We'll open-source the details piece by piece over the coming weeks.

Streaming the run: https://t.co/ZSxahzJRju

[Dario Amodei @DarioAmodei]

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.

Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.

You can read the full post here: https://t.co/OGyPb7yaYt

[Albert @yangAdam9]

@DarioAmodei Deceleration only works if everyone decelerates. With Chinese competitors in the race, you can't slow down and you can't scale globally. So play the endgame carefully — you must win. For humanity.

[Albert @yangAdam9]

再一点则是针对index sparse attention做了大量复用 ，除复用kv cache ，同样也做了近临index复用

[Albert @yangAdam9]

Deepseek v4.1并不是回归了encode-decode注意力。 它只是区分K V cache的重用层。 简单点说，全部采用采用top-k sparse attention +slide window attention 512+128去做真正的softmax 点乘。不同的从20层开始，k v cache 全部重用，只有query依然每层独立计算结果。

[ClaudeDevs @ClaudeDevs]

https://t.co/BAGbeyTkga

[Jensen Huang @JensenHuang]

@ChaseLochmiller @OpenAI GPT-6 Astra, trained on ~100K+ NVIDIA Grace Blackwell NVLink72. From ChatGPT to o1 to Astra in 4 years.

AGI has arrived. Congratulations @OpenAI team.

400K GPUs coming online next.

[Bleys Goodson @bleysg]

It is a 2 to 4T param model. They are serving it across 70-100 wafers. To get healthy serving characteristics, they are essentially putting at most one layer per wafer, and the model is in the ballpark of 70-90 layers.

There's a couple of different ways this could be served and model sizes implied by that. One is if they keep the heavy KV caches they've used before. Another is if they go with lighter KV cache designs more akin to DeepSeekV4 or Hybrid SSM models.

The fact that they've partnered with Cerebras and designed with the hardware in mind means they're much more likely to have gone the second route. That SRAM bandwidth is too precious for a heavy KV cache. As such, something like the below is the center of probability mass: 3T total, 150B active, 70 layers.

[美国驻华使领馆 U.S. Mission to China @USA_China_Talk]

卢比奥国务卿：古巴的共产主义政权权贵们统治着一个失败的国家：古巴老百姓忍饥挨饿，而卡斯特罗家族及其他政权内部人员却通过规避制裁及其他非法手段中饱私囊——并企图将颠覆性的马克思主义意识形态输出到美国和西半球的其他国家。今天，我已将劳尔·卡斯特罗的孙子菲德尔·埃内斯托·卡斯特罗·卡利斯、古巴外贸银行以及隶属于腐败的卡斯特罗金融和情报网络的四个实体列入制裁名单。特朗普总统和我对古巴终将获得自由的承诺坚定不移。https://t.co/Xh7wgjC5XG

[Tibo @thsottiaux]

We will give one banked reset for every day you don't have access to Astra on your paid ChatGPT plan, starting today. Team is moving mountains to give access as fast as we can.

First one will land in ~ 3 hours. There is still time to create your account if you don't have one.

[Edward Hu @edwardjhu]

How does one RL post-train a 397B model for long-horizon knowledge work? 👩‍💼

We share every step we took to bring Qwen 3.5 397B from 16.1% Pass@1 to 27.3% on APEX-Agents using DPPO, including final models weights and the full training script🚀 This is the first of many works from Mercor Research on open model training research.

Full blog: https://t.co/GWgR2DB7iy
Source code: https://t.co/96h8ADfvBp

[Sebastian Raschka @rasbt]

A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".

It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.

About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."

Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.

In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.

In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.

Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)

While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.

In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.

Also, the statement "the new technique works in a way that obscures some or all of the AI's reasoning, otherwise known as 'chain-of-thought'" is not necessarily true with respect to the looped transformer method. It's possible that The Information journalist refers to some other technique or misunderstood the looped transformer method.

Reusing layers does not by itself suppress visible chain of thought. It adds computation in hidden states before the next token is emitted, just as ordinary transformer layers do.

But based on the information we have, the only plausible interpretation here is that if a model uses more of these recurrent passes, it may need to generate fewer intermediate reasoning tokens. So then more of its computation happens in latent activations that cannot be read as text. But we would get the same effect if we were scaling up the model size, like GPT 5.6 Luna -> GPT 5.6 Sol.

[Albert @yangAdam9]

I’m skeptical of looped Transformers. Unconstrained latent recurrence may drift off the reasoning manifold learned during pretraining. Explicit CoT’s language bottleneck may be a feature, not a bug: a periodic anchor and error-correction step.

[Jakub Pachocki @merettm]

I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.

OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.

[Tao Wang @Tao100086]

接上一篇，张小珺与谢青池的访谈，谢读了 200 多篇 AI 论文，从中挑出 36 项经典论文，我让Codex整理出下载地址，放在这里分享给有需要的人。

完整节目与 Show Notes：
https://t.co/5yDzK8Z42k

他读论文的方法，带着这几个问题去读：当时遇到了什么问题？旧方法为什么不够？新工作解决了什么？它又为哪些后续技术铺了路？

下面按 4 条历史主线整理。建议第一次阅读不必逐行推公式，先看摘要、Introduction、核心图和 Conclusion，把因果链串起来。

一、模型范式的变迁
这条线回答：神经网络为什么变深、Attention 为什么出现，Transformer、MoE、推理与 Agent 又从哪里来？
Brook for GPUs（2004）
https://t.co/ZLA11RXJkp
AlexNet（2012）
https://t.co/oPXZAv8OCj
Sequence to Sequence Learning with Neural Networks（2014）
https://t.co/dAHO1aKGQ0
Neural Machine Translation by Jointly Learning to Align and Translate（2014）
https://t.co/ENt59aIzb6
Distilling the Knowledge in a Neural Network（2015）
https://t.co/oZN9rPscxT
Deep Residual Learning for Image Recognition / ResNet（2015）
https://t.co/pCxm1aPLi0
Sparsely-Gated Mixture-of-Experts（2017）
https://t.co/A4AyNTsij5
Attention Is All You Need（2017）
https://t.co/3zqAptnMCs
AlphaGo Zero（2017）
https://t.co/vgw4lq1sUV
The Bitter Lesson（2019，文章而非论文）
https://t.co/ULQr8DIVei
LoRA（2021）
https://t.co/bxapJx5RMM
Chain-of-Thought Prompting（2022）
https://t.co/dKClVRvrCd
ReAct（2022）
https://t.co/K3ZpAcmP0q

二、Infra 与数据
这条线回答：模型怎样从单机实验扩展到万卡训练？模型、数据和计算预算应该怎样匹配？
ZeRO（2019）
https://t.co/E3SgPiaYSA
Scaling Laws for Neural Language Models（2020）
https://t.co/kk2SquAoXi
Training Compute-Optimal Large Language Models / Chinchilla（2022）
https://t.co/rLEjTwua4V
LAION-5B（2022）
https://t.co/NlYzev7F6j
The RefinedWeb Dataset for Falcon LLM（2023）
https://t.co/0qPVjwGm4Z
MegaScale：将大模型训练扩展到 10,000+ GPU（2024）
https://t.co/IJHTJCdReh

三、语言模型的发展
这条线回答：词怎样变成向量？预训练为什么有效？模型怎样从任务微调走向上下文学习，再变成遵循指令的助手？
Word2Vec（2013）
https://t.co/tQoF0Ye2Zj
Google Neural Machine Translation / GNMT（2016）
https://t.co/oz3g68NONA
GPT-1（2018）
https://t.co/e3sV5ahucY
BERT（2018）
https://t.co/nvYjpJ39rs
GPT-2（2019）
https://t.co/XcSMTkowob
GPT-3（2020）
https://t.co/WWrr6bKLsq
InstructGPT（2022）
https://t.co/oYIFtoxgQJ
Tulu 3（2024）
https://t.co/6pc8yOT5if

四、多模态模型的发展
这条线回答：深度学习怎样进入视频和图像生成？Transformer 又怎样扩展到视觉与扩散模型？
Large-Scale Video Classification with Convolutional Neural Networks（2014）
https://t.co/TCQI85EjhS
Two-Stream Convolutional Networks for Action Recognition in Videos（2014）
https://t.co/S7oirKI6tm
Generative Adversarial Networks / GAN（2014）
https://t.co/fC44lv4alM
Deep Unsupervised Learning using Nonequilibrium Thermodynamics（2015）
https://t.co/KKuolQ0XtC
Denoising Diffusion Probabilistic Models / DDPM（2020）
https://t.co/SirX0txbAm
Vision Transformer / ViT（2020）
https://t.co/qL5vKZ6Qrw
CLIP（2021）
https://t.co/MIh9YUXOzI
Latent Diffusion Models / Stable Diffusion 的技术基础（2021）
https://t.co/4pQnD9antU
Diffusion Transformer / DiT（2022）
https://t.co/sVjqpwIfNw

两项补充阅读
它们在访谈中被明确提到，但不属于正式 36 项主清单：
The Hardware Lottery（2020）
https://t.co/3keB0pZVjl
PPO（2017）
https://t.co/gjuyA1ydbo

如果时间有限，可以先读这 10 项：
Attention Is All You Need、GPT-1、GPT-2、GPT-3、Scaling Laws、Chinchilla、InstructGPT、LoRA、ReAct、CLIP。

[Percy Liang @percyliang]

Marin 535B-A23B is ~7% done training, and so far things look on track. Next Tuesday (Sept 1 @ 10 PT), we will have a Zoom panel/discussion where the Marin team will talk about the design decisions that went into this run, the tradeoffs made, and our learnings. If you're interested, join the Discord for details (https://t.co/xwJSxd2IxH).