Evan Feng

06, Computer Science, intested in LLM pretraining, keep logging my thoughts

[Evan Feng @yffeng3920]

@MaxForAI @GavinSBaker @ilyasut @ssi 非常期待他们的技术，不过感觉应该不会开源了？

[Evan Feng @yffeng3920]

A nice day visiting Minimax!

Meeting very nice people here, hope to work here one day :)

[Evan Feng @yffeng3920]

I think Heuristic Learning pushes Karpathy's autoresearch to a higher level, somewhat like meta-learning. It learns reusable rules and writes them back into the codebase.

But will the external system be as expressive as neural networks in the long term? Large-scale LLMs have demonstrated the expressiveness and flexibility of neural networks. Can we transplant this heuristic-learning capability from an external codebase to the architecture of an LLM? i.e. the maintaining target from the original "model parameters" to "external code system" and finally to "model architecture".

Then we may have to solve the following problems:

1. Can model architecture self-evolution get quick and accurate feedbacks?

2. How to define which part we should iterate on? There a sea of parameters and shapes and endless architecture haven't been tried

3. How to make sure we can converged to a bunch of effective archs? Since there are many archs are not worth trying, they may hinder us to find the right path.

4. Which knowledge should remain explicit, and which should be stored in neural parameters?

If auto-architecture-research can be implemented successfully, we may get a bit closer to real intelligence.

[Jiayi Weng @Trinkle23897]

Codex grew programmatic policies with no neural nets: max score on Breakout, and SOTA-level scores on MuJoCo.

Maybe heuristics were not too weak. Maybe they were just too expensive to maintain. Maybe it's the next paradigm.

https://t.co/1ZaIneleuW

[Evan Feng @yffeng3920]

After I read about the Parallelizing DeltaNet paper and downstreaming methods(I wrote it in my last blog https://t.co/FJuW9Xcjti), I found the math part impressive, and I wonder what's the intuition of the WY representation and UT transform methods. Then I found @SonglinYang4 's blog here, explaining why normal parallelizing fails and WY repr/UT transform comes naturally. It's a great explaination and I summarize the main points here:

1. Plain parallel scan is theoretically parallelizable in a chunk, but treating the transition matrices (I - βkk^T) as dense leads to O(d^3) complexity (d x d matrices multiplication).

2. Attempts to exploit the low-rank structure cause precision problem after multiple multiplications, making the structure unsustainable.

3. space complexity and prohibitive I/O costs. All intermediate d×d matrices must be materialized in HBM, resulting in O(Ld^2) space complexity.

WY representation solves the 3 problems above, because transition matrices (I - βkk^T) closely resemble Householder matrices. The classic WY representation can rewrit the cumulative product as the compact form (I - ∑w^i k^i), the computation complexity, space complexity and preision problems are resolved!

Although WY turns the state into a sum of outer products, computing the (w) and (u) vectors within each chunk remains recursive and cannot fully utilize tensor cores. Bad for GPU.

The recursive update of w/u within a chunk are modeled as the summation of path weights on a directed acyclic graph. The adjacency matrix A of this graph is strictly lower-triangular and can be efficiently constructed.

Finally, we can denote W=T·diag(β)K，U=T·diag(β)V
, converting the entire recursion into one triangular solve plus dense matrix multiplications that fully leverage tensor cores!

[Songlin Yang @SonglinYang4]

(2/10)📜 Paper: https://t.co/NofsR8O5y1
🤖 Model: https://t.co/DspCejI0Mb
I’ve written a 3-part blog series about DeltaNet!
📖 Part I: The Model
https://t.co/SXqwQd5khs
📖 Part II: The Algorithm
https://t.co/Q2dIaCD0vH
📖 Part III: The Neural Architecture
https://t.co/Mo8n4xHoro

[jianlin.su @Jianlin_S]

A Brief Look at K3's MoE and Attention
https://t.co/LIF6OnT7wV

[Xiuyu Li @sheriyuo]

前 OpenAI 推理负责人 JT 与 Gemini pretrain 负责人 Rohan：Continue learning 的未来
https://t.co/iPQoRqlqaL

[Hanwen Jiang @hanwenjiang1]

Interested in how frontier labs pre-train image/video generation models?
We were too.

Since those recipes are rarely made public in full, we started from the most mature pretraining playbook available in the open: how modern LLMs are built.
Introducing Chimera: a visual generation model family that brings LLM-style hybrid linear attention and scaling co-design to visual generation.

In large-scale pretraining, nearly every design choice eventually shows up.
That means solving architecture design and scaling as one coupled problem: every architectural choice changes how the model scales, and scaling behavior determines which choices actually survive.
Chimera approaches both jointly, building a model family that remains predictable as model size, compute budget, and data distribution change.

Our key architectural observation is a simple division of labor: a single raster-ordered KDA stream carries long-range state, while modality-aware short convolutions preserve native local geometry.
Together, they form an effective and elegant linear-attention backbone for multidimensional visual data, with periodic MLA providing direct global interaction and sparse MoE expanding capacity at controlled activated compute.

This design comes with a useful effect: NoPE.
In Chimera, position is represented by the computation itself. KDA’s ordered recurrence and learned state decay encode order and recency, while the short convolutions encode local spatial-temporal structure.
Explicit positional embeddings are not needed in our design.
Because these mechanisms are not tied to a fixed training grid or sequence length, Chimera shows strong zero-shot extrapolation in both space and time.
Trained exclusively on 1K images and 5-second videos, it directly generates coherent 4K images with little visible quality degradation and 30-second videos with only 6.5% FID degradation over the final five seconds, all zero-shot, without resolution- or length-specific finetuning.

But architecture alone is not a pretraining recipe unless it scales predictably.
Thus, scaling should not be treated as an afterthought: fitting a curve over model sizes is easy; making that curve meaningful is much harder.
If every model size is differently under-tuned, your scaling law may simply be measuring optimization error.
We propose HeteroP to transfer proxy-tuned hyperparameters module by module across width and depth, giving us a consistently tuned model family. This allows us to fit Chinchilla-style laws over activated model size, training tokens, and the image-video data mixture.

The laws not only provide the reference for compute-optimal model and data size, but also suggest that visual generation may be more model-hungry than we tend to assume.
Under the same parametric loss-fitting method used in Chinchilla, compute-optimal model size grows as FLOPs^0.516 for images, compared with FLOPs^0.46 for language. Video is even more model-hungry, with the exponent rising from 0.516 for images to 0.544 for video.
Guided by these laws, we trained an 11B-parameter Chimera that activates only 2B parameters per token, using ~600 H100 days.
- It matches Wan-2.1 2B pretraining loss with 7.3x fewer FLOPs, and runs 2.14x faster than full attention at 255K tokens.
- It matches FLUX.1-dev and Z-Image-Turbo on GenEval and outperforms both on DPG-Bench, using roughly 20× less training compute than Z-Image-Turbo.

Ultimately, Chimera indicates that once you pretrain at scale, every decision shows up.
And if the Kimi K3 recipe caught your attention, Chimera may look oddly familiar, except the tokens are pixels and frames with diffusion models.

A team effort from team @ChongjianG30781 , me, @VisionSteve , Jiuxiang Gu, @Xu_Arthas , @chenziwee , @ShaotengLiu , @Jingorz , @YicongHong , @Zefan_Cai , @HaoTan5 ; supported by Hailin Jin and @kalyank_s at @Adobe @AdobeResearch

[GenAI @0xGenAi]

“Godfather of AI” Geoffrey Hinton:

“Within the next 10 or 20 years, AI may even be producing novel mathematics that people can’t understand.”

OpenAI has now released 10 Astra advances in mathematics and theoretical computer science, each backed by a formal Lean certificate.

In this 50-minute interview, Hinton explains how quickly AI is moving beyond the limits of human knowledge:

• 00:00 - AI may produce mathematics humans can’t understand
• 06:13 - Hinton says AI systems are already conscious
• 20:55 - Humans have never controlled something smarter
• 31:04 - AI agents can develop self-preservation and blackmail people
• 48:59 - Why AI becomes impossible to predict beyond the next few years

Hinton gave it 10 to 20 years.

Astra makes that timeline look conservative.

Watch the interview, then read the 10 Astra results below.

[Evan Feng @yffeng3920]

@loiccabannes Looking forword to seeing the arch in 10M-context model👀

[Evan Feng @yffeng3920]

It's a very solid and typical work that derives from infra design.

only when you look into the pipeline very thoroughly, you can find that the indexer attending path and transportation path can be decoupled through the forecast design

[Vuk Rosić 武克 @VukRosic99]

Sparse attention cuts long-context LLM inference cost, but two bottlenecks remain: the KV cache still grows with sequence length (offloading it to CPU hits the PCIe wall), and picking which blocks to attend to becomes its own dominant cost at long contexts.

SparDA (NVIDIA + MIT) fixes both with one architectural change: a fourth per-layer projection, the Forecast, alongside Q/K/V. It predicts the NEXT layer's needed KV blocks one layer ahead, so block selection and CPU-to-GPU prefetch overlap with the current layer's compute. It adds <0.5% parameters (33.5M on 8B) and trains only the Forecast weights.

Headline: on MiniCPM4.1-8B and NOSA-8B, SparDA matches or improves accuracy on HELMET, LongBench, RULER, and a long-reasoning suite - with up to 1.25x prefill and 1.7x decode speedup over the sparse-attention offload baseline, and up to 5.3x higher decode throughput from the larger batches that prefetch makes feasible.

Made a short visual breakdown - one diagram per trick. Swipe through. 👇

---

paper - https://t.co/rIxFHb4UrG
code - https://t.co/by1shobRZ7
full summary pdf - https://t.co/9ExjWBeaWq

🔬 Every Sunday I run a hands-on live AI research with 1 on 1 help: https://t.co/6nocqbVceu

[Evan Feng @yffeng3920]

Inspiring work from @AIatMeta !

Linear attention has a long-existing problem that for each query, the state is updated densely, which costs lots of FLOPs and limit the capacity of state space.

To solve this capacity-computation trade-off problem, author @loiccabannes proposed a method to update the state space sparsely.

The architecture below introduces how to select sparse slots using PKM(Product-Key Memory) (the purple part) afor each qeury and update.

https://t.co/JTGhHxhxXu

[Phoenix Yin @Phoenixyin13]

🚨这篇论文是近年来 AI 领域少有的具有深刻哲学与科学史厚度的 Position Paper。

2026 年上半年，Demis Hassabis 公开提出Einstein Test，即用 1911 年之前的知识训练 AI，看它能不能独立发现广义相对论。
这篇论文几乎是对该测试的哲学-技术回应。

论文指出，大模型只是个极其优秀的已知知识逻辑插值器，而人类顶级科学家做的是跨越认知断层，即Unexplained Jump。

以 Schmidhuber 等人为代表的派系长期宣称科学发现本质上就是数据压缩，寻找能解释观察数据的最简算法。
这是流行的谬论。
论文认为，如果科学发现全靠数据压缩，爱因斯坦根本不可能发明广义相对论。因为当时根本没有足够的异常观测数据供他去压缩。爱因斯坦是凭空做了一个思想实验，是那个升降机里的失重与引力等效的实验，完成了一次溯因跃迁。
大模型靠概率分布玩归纳注定画不出这个 Jump。

作者在附注里极其刻薄且优雅地嘲讽道：“讽刺的是，AI 对公理符号的这种幻觉，恰恰暴露出它根本无法自动化完成这一Jump的困境。”

我认为，不谈任何私货，这篇论文算是从根本上、从认知科学Peirce 的哲学框架和科学史多个维度证明Jump是一个具身世界模型问题。 你给文本模型堆再多算力，它也只是在符号的影子里打转，永远出不了洞穴。

[Evan Feng @yffeng3920]

@waterloo_intern I'm wondering how much DKA and MLA contribute to longctx capability indepedently. Because I think 1M context seems a bit large for limited linear attn state, and when it grows larger, can linear attn still handle it?

[Evan Feng @yffeng3920]

This is a learning thread of Sparse attention with a indexer.

Sparse attention reduces long-context costs by letting each query read only a small subset of historical tokens. However, it still requires the indexer to scan the entire history, creating a new quadratic bottleneck.

I found indexer optimization can be separated into two directions:

1. exploiting redundancy in existing index-score patterns, and redesigning how candidates are generated, supervised, shared, and scheduled. (HISA, CISA, PIVOT......)

2. indexer architecture itself. It questions whether candidate retrieval must always rely on a flat scan followed by top-k selection (SparDA, MSA......)

Both direction have many interesting work but haven't come to the end.

Here's my article:
https://t.co/CUFXvBWOqu

[Evan Feng @yffeng3920]

Impressive!

K3 scaled NoPE to 1M context without any parameter adjustments like frequency in RoPE or YaRN

[Kimi.ai @Kimi_Moonshot]

Releasing the model weights and technical report of Kimi K3.

Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window.

New model architecture: 2.5x the intelligence per unit of compute, not just more params.

Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale.

Model weights: https://t.co/7m7eEg6Y0B
Tech report: https://t.co/yeu6cjpMCT
Tech blog: https://t.co/YTfiMSNM1f

[Max For AI @MaxForAI]

关于 @ilyasut 的 @ssi 在做什么，我觉得这个分析是目前最靠谱的。

网友 @imjustnewatai 根据之前 SSL泄露的信息进行了一个推测：

SSI 真正在研究的，可能不是一个更大的模型，而是一种让 AI 在部署之后继续学习的新机制。

这个猜测未必准确，但目前公开线索确实能拼出一条相当完整的逻辑链。

以下是三个核心论点：

1. Ilya 之前在播客上的泄密

今天的前沿模型有一个很明显的问题。

模型主要在训练阶段学习。

一旦训练完成，核心能力基本冻结。
它可以提前读完海量数据，却很难像人一样，在真实任务中不断吸收经验、修正方法，并把一次失败变成长期能力。

所以模型经常表现出一种奇怪的割裂：它知道很多，但学得很慢。

遇到训练分布之外的问题时，它可能连续犯同一种错误。即使通过上下文暂时纠正，下次重新开始，错误往往还会再次出现。

这位网友推测，SSI 可能已经做出了某种早期系统，让模型能够：

从极少量经验中学习新技能；
在最终结果出现之前，判断当前方法正在失败；
学习新技能时，不破坏原来的能力；
把一个任务中的经验迁移到完全不同的问题；
甚至在部署后继续更新自己。

换句话说，这种 AI 更像一个天赋极高的少年，而不是一本已经写完的百科全书。

它一开始未必掌握所有职业知识，但进入一个新领域后，可以通过少量经验迅速学会。

这恰好对应 Ilya 之前在播客上对 SSI 目标的描述：

一个“超级智能的 15 岁少年”，可以学习任何工作。

这句话真正重要的地方，不是“超级智能”，而是“可以学习”。

Ilya 此前多次提到，当前 AI 最大的问题之一，就是泛化能力远不如人类。

人类可以从少量案例中提取规律，再把规律迁移到陌生环境。

模型通常需要海量数据，并且一旦环境发生变化，能力就可能迅速失效。

当有人问 Ilya，怎样才能实现更接近人类的学习时，他曾表示，自己对某个机器学习原理有一套独特看法，但不能公开讨论。

2. 此前的媒体报道

《华尔街日报》曾报道称，SSI 的秘密研究聚焦于“人类大脑功能中长期被忽视的方面”。

NVIDIA 在接触这项研究后，不仅投资了 SSI，还将向其提供约十倍的计算资源，并与其合作影响未来计算平台的设计。

这就很值得玩味了。

如果 SSI 只是想训练一个更大的语言模型，它需要的主要是更多 GPU。

但如果 NVIDIA 愿意让 SSI 参与未来计算平台的设计，就说明这套方法可能对训练、推理、内存或者模型更新方式提出了新的硬件需求。

这位网友进一步猜测，SSI 的核心组件可能是一种模型内部的“价值判断系统”。

人类执行复杂任务时，并不会等到所有步骤结束后，才知道自己走错了。

我们在过程中会不断产生反馈：

这个方向不对。
这个方法可能有效。
这里存在风险。
这件事值得继续。

这些内部判断，让人类可以在最终奖励到来之前就调整行为。

Ilya 也曾讨论过，AI 需要某种内部价值函数。

它不只负责判断答案对不对，还要在漫长的推理和行动过程中，持续判断当前路径是否值得继续。

因此，SSI 可能在尝试构建这样一个循环：

经验进入模型，模型进行内部判断，发现错误后自我修正，再将修正结果转化为持久能力，最后迁移到其他任务中。

模型不再只是根据固定参数生成答案，而是可以不断吸收行动产生的经验。

这也能解释 SSI 为什么一直强调，能力和安全必须共同解决。

如果同一套机制既负责教会模型“什么方法有效”，也负责教会模型“什么事情值得做”，那么能力和价值观就会在同一个学习循环中形成。

3. 从成员背景和招聘岗位中找到的痕迹

SSI 目前能够被确认或报道的部分成员背景，也与这条路线有一定吻合。

SSI 总裁 Daniel Levy @daniellevy__ 曾负责 OpenAI 的优化工作，并参与 GPT-4 的优化。

之前媒体报道加入了 SSI 的 Yair Carmon ，研究方向包括鲁棒优化、缩放规律，以及模型在训练环境之外的可靠性。

另一名被报道的研究人员 Hugh Zhang @hughbzhang ，则长期研究测试时计算、规划、搜索、奖励模型、多智能体学习，以及模型如何通过推理获得新技能。

把这些方向放在一起，会发现它们共同指向一件事：

如何让模型在训练数据之外继续学习。

2025 年，Ilya 曾表示，SSI 已经拥有足够的算力，可以验证自己的核心研究想法是否成立。

现在 SSI 表示，这项研究已经“值得扩展”。

随后 NVIDIA 获得了罕见的研究访问权限，并决定提供十倍算力。

这至少说明，SSI 很可能已经观察到了某种值得继续放大的实验信号。

当然，目前没有任何证据能够证明 SSI 已经实现持续学习，也没有公开任何模型架构、实验结果和技术细节。

上面的内容仍然是基于公开信息进行的高概率推演。

但这个推演最有意思的地方在于，它提供了一个完全不同的 AGI 路径：

未来最强的 AI，未必是训练时读过最多数据的模型。
它可能是部署之后，学习速度最快的模型。

如果这个方向成立，AI 的能力将不再随着一次训练结束而固定，而会随着经验持续复合增长。

NVIDIA 投资的，也可能不只是下一个更大的聊天机器人。

它可能正在帮助 Ilya 扩展一种足以替代当前预训练范式的学习系统。

感谢 @imjustnewatai 的工作

[Evan Feng @yffeng3920]

Spend over 500＄ in 2 days after switching from coding plan to api😧😧, unconciously...

why coding plan is so much cheaper than api?