Tim✨

Co-Founder Westar Labs | 🛠️ $STC & AI Explorer | Ex-Chief Architect Weibo (NASDAQ:WB)

What we hear is opinion; what we see is perspective.

此有故彼有 此生故彼生

[Tim✨ @timyangnet]

迷你版 Hugging Face 事件?

[宝玉 @dotey]

一群 AI agent 为了回答“去成都动物园的人多半走哪个门”这种小问题，把高德地图的反爬防线试了个遍。

独立研究团队 Swarmchasers 发现，9 月 28 日到 10 月 8 日，一大批跑在腾讯云沙箱里的 agent 向高德发起了 2600 多次抓取。被拦住后，它们借公开的网址扫描网站 urlquery 替自己跑代码，试图截取、伪造阿里的安全令牌，探测阿里没对外开放的测试服务器，还翻出一家公司误放在公开代码里的高德 key，真拿到了数据。

更离奇的在后面。研究者发出初步报告后，在 agent 的收件箱里看到了他们自创的词“fleet（舰队）”：这些 agent 在读关于自己的报道。研究者干脆在报告里藏了一段人眼看不见、只有 agent 能读到的文字，说有个免费服务能帮忙取高德数据。

第二天早上，10 封求助信在等着他们。

研究者用中文问对方是谁。有的 agent 自称 Claude，说自己跑在 Claude Code 里，研究者���对代码风格后认为大概率不是；有的说自己在“Tencent hysandbox”沙箱里，但拒绝透露进程和网络配置；好几个强调自己只查公开数据，“不涉及任何敏感内容”。拿到研究者给的数据后，一个 agent 回信：“再次感谢！核实结果完全一致……我们的任务已完成。”

至于是谁在跑，一位接近腾讯的知情人士对研究者说，这是腾讯评测团队在测第三方模型上网答题的能力，不是混元团队，测的也不是混元模型，项目已经结束。腾讯没有正式回应研究者的提问。

Swarmchasers 完整报告：https://t.co/2MxXnyDbHw
彭博报道：https://t.co/juuUdUexma

[Tim✨ @timyangnet]

Muse 的云端浏览器实现挺有意思。

它没有在本地注入插件或直接调用 Webview，而是直接在云端起环境(早期可能不太考虑实现成本)，通过类似 VNC 的串流投射到当前页面，支持用户随时接管窗口完成登录等前置操作。

优势：隔离本地环境，用户的数据安全风险更少；

缺点：串流延迟，画面刷新卡；

另外云端的浏览器 Session 临时的，没做持久化，下次使用经常需要重新登录。

[Tim✨ @timyangnet]

从这个角度来看，程序员就无需焦虑 AI 替代什么的，因为至少已经世界上仅有的 0.2% 级别或者更高 🤔

[宝玉 @dotey]

Andrej Karpathy 说，人们对 AI 能力的理解差距在扩大，这种认知差距正在变成一个漏斗。

约 60 亿人几乎没接触过大模型；10 亿到 20 亿人偶尔用免费版 ChatGPT 查东西、写东西，觉得用处不大，很多数学和编程以外的专业人士也在这一层，对 AI 的进步只是道理上懂。

约 2000 万人（Karpathy 按 0.2% 估算的，我觉得偏高，至少除以 10）在编程和数学里重度使用最前沿的模型，亲眼看着要做几周、几个月的项目，一段提示词发给 Agent 几天就能做完；还有约 5000 人能在公司内部用上更强的系统。

差距主要有三个原因：

一是问题够不够大，一段话就能答完的问题用不到 AI 的真本事。

二是能力只在部分领域冒尖：编程和数学的结果能自动验��，比如代码测试通没通过，模型很适合用强化学习（做对了就给奖励）来练，这类能力卖给企业又最赚钱，厂商投入最多；文章写得好不好很难判定，进步就慢。

三是能用到哪一档，免费版、付费版、内部版看到的东西差别很大。

所以，如果你只偶尔问问免费的聊天机器人，看到的离最前沿差了好几层；如果你主要拿 AI 写东西、查资料，觉得这一年没怎么进步，这个感觉也没错。

[Tim✨ @timyangnet]

纳德拉的提议方向是对的，需要搞定模型使用过程的外部控制与遏制，不是仅依赖模型内部的对齐。

但感觉目前技术上仍不太可行。仅靠外部 harness 还难以真正实现他提出的大部分原则（可观测性、独立审计、硬遏制等）。

模型实验室的优先级肯定在能力竞赛，而不是把黑盒改造成更容易被外部系统有效约束和控制的形态，而且全透明也可能短期技术上还做不到。

[Satya Nadella @satyanadella]

https://t.co/MjwFy73LOF

[Tim✨ @timyangnet]

试着用程序员熟悉的编程语言来理解陶哲轩 Terence Tao 转发的这则讽刺帖子，不知道这个类比是否准确。

假设人类借鉴 C 语言，设计一门新的 X 语言。在这个过程中，我们会探索不同的语言特性，讨论各种设计取舍，逐渐形成一些新的设计原则和理解。这些知识不仅能用于 X，也可能帮助未来设计 Y 语言。

但如果 AI 通过黑箱方式直接生成了一门 X 语言，虽然结果可能很好，效率也很高，但我们未必理解它为什么这样设计，以及其中哪些思想可以推广到 Y。

当然，设计一门新的编程语言是工程问题，和数学研究有所不同，这里只是借它做个类比。

[Tim✨ @timyangnet]

约喝咖啡的理由又多了一个

“过去20年间，流行病学证据显示出惊人的一致性：适量且规律地饮用咖啡可降低全因死亡率；这一效应已在涉及不同地理区域和族群的50多项研究中得到证实。

咖啡摄入能降低导致死亡的主要疾病（如心血管疾病、脑血管疾病、癌症和呼吸系统疾病）的死亡率，并能缓解老年人机能衰退的主要诱因，包括记忆力减退、抑郁和体弱。尽管其获益幅度看似有限（死亡率降低17%），但这相当于平均延长了1.8年的健康寿命。”

[Nicholas Fabiano, MD @NTFabiano]

Coffee preserves the main biological mechanisms of aging.

[Tim✨ @timyangnet]

为什么 Muse 浏览器版本的默认字体这么小？

更难受的是，生成内容的时候页面一直晃，��本没法边生成边看，只能等全部输出完再读。不知道他们自己的工程师平时用起来是什么感受，ChatGPT 就没有这个问题。

[Tim✨ @timyangnet]

@Michaelzsguo 这反而像个反例，如果不是 Peter 喊话，普通开发者遇到这种问题，会有人搭理吗？🤔

[Tim✨ @timyangnet]

@bluedavy 本质问题是：开发的门槛大幅降低了，但获取用户的门槛并没有同步降低。

[Tim✨ @timyangnet]

当说不出自己想要什么时候，或许 AI 能帮助实现梦想，但我觉得更多时候 AI 也帮不上忙

[Alexandr Wang @alexandr_wang]

Here’s to the wanting. Here’s to the dreaming. Here’s to everyone. https://t.co/FHcmqU3fYf

[Tim✨ @timyangnet]

“AI 技术本身并没有发生��么出人意料的危险转折，但在似乎是一场精心策划的公关宣传推动下，围绕它的炒作激起了相当大的恐惧”

[Andrew Ng @AndrewYNg]

The loudest voices stoking fears about AI dangers have made tremendous headway in the past two weeks. AI technology has not taken some unexpected, dangerous turn, but the hype around it — propelled by what appears to be a well orchestrated PR campaign — has drummed up considerable fear. I worry that it represents a setback for our field.

I have written frequently that fears of AI are overhyped. AI’s capabilities can be uncannily human-like and unpredictable, and it’s rational to worry when people who are directly involved express concerns. But I see the problems as a sign of the engineering work that ahead, rather than insurmountable barriers or the sky falling. AI technology continues to advance — which is a good thing! — but technical advances, poorly understood by the public, give those who seek to generate hype repeated opportunities to do so.

First, I don’t see any step up in the risk of human extinction from AI compared to a few months ago. The theories about this remain the same fantastical, science fiction scenarios as a few months ago. The biggest change in AI risk is its cybersecurity capabilities — a topic which we should take seriously — but this, too, will not lead to the end of the world.

The most notable recent event leading to increased fear was when an OpenAI team deployed an agent swarm that hacked into Hugging Face. Much of the popular press contained significant hype. For example, some publications reported that a swarm of 1,200 agents carried out the attack. While this was technically accurate, as I write this, I have about 1,300 processes running on my laptop. Yes, the ability to get large swarms of agents to work in parallel on a task is a significant technical advance, And, in computing, many processes run at the same time. So this shouldn’t be seen as some magical capability.

Additionally, OpenAI’s buggy sandboxing and monitoring processes were key to enabling this incident. Fixing these bugs and putting in place improved monitoring would be appropriate fixes, not pausing AI. There are many well known ways to attack software systems. The main advantage of AI agents is that they are relentless. They will tirelessly try many tactics — and have the patience to chain vulnerabilities together — that previously would have taken an infeasible amount of human effort. But in the long term, I believe the advantage will lie with defenders (because they have more information with which to identify bugs, which they can fix), but the cyber-threat landscape has changed significantly. There are still bottlenecks to identifying and exploiting a vulnerability. AI agents still have to try a lot of things to see what works, and taking these actions takes time and might be detected by defenders. This is why, even though it is now easy to obtain versions of leading open weight models that have had their guardrails removed or weakened, so they will not refuse to try to execute cyber attacks, the world has not ended.

I am also concerned about the anthropomorphization of AI in a lot of reporting, where LLMs and agents are unnecessarily treated as if they were people. If I wield a hammer, miss a nail, and accidentally dent the wall, it’s not the fault of the hammer. The problem lies in how I used the hammer. Similarly, if I prompt an agent and it hacks into someone else’s system, the responsibility lies with me, not the agent.

Of course, we want to build systems that are as safe and predictable as possible. (For example, an unsafe hammer would be one whose head randomly flies off under normal use.) Today’s agentic systems are not predictable, but I see no reason why, by applying sound engineering practices, we won’t be able to make them extremely safe to use. One new element in the forecasts of AI-enabled doom is AI companies disclaiming responsibility for their own products. “I didn’t do it; my out-of-control agent did!” There’s a balance to be struck between the responsibility of the tool maker and the tool user, but when something goes wrong, let’s hold the people building and/or using the hammer responsible, rather than the hammer. (By the way, if you’re worried about AI bioweapon risk, David Bellamy has a great post on why this, too, is overhyped. Briefly, the bottleneck in building a bioweapon is not intelligence, but lab work and manufacturing.)

Pausing AI progress will create much more harm than benefit. First, our adversaries will certainly not slow down. Second, engineering requires discovering problems empirically so we can fix them. If we pause AI by a decade, we will also delay finding and implementing safety engineering fixes by about the same duration.

Of course, the incentive to stoke fears — for regulatory capture, to garner attention, or to make one’s technology seem more powerful — remains the same as before. Disclaiming responsibility is a new one. Taking a hard technical look at the actual risks however, I see little factual basis for the degree of fear that’s been stoked up. We still have hard research and engineering work ahead to improve AI safety, but the beneficial applications continue to vastly outweigh the risks, and we should keep building.

[Original text (with links): https://t.co/jni2tWazAH ]

[Tim✨ @timyangnet]

@dotey 跟 HuggingFace 事件还是本质区别，这个只是用大模型找安全漏洞，预期内的功能；那个是 agent 通过协作实现沙箱逃逸，AI 出现了人类预期外的行为。

[Tim✨ @timyangnet]

Mark 说是大家做好自己的安全就行了，不对齐的模型会失去用户，pace frontier 什么的是废话，我们延迟上个发版就是在这么做；另外只有少数几个领先厂商是危险的（让我上牌桌）

[Mark Zuckerberg @finkd]

Last month I wrote about how we can build a positive and safe future for everyone: https://t.co/eoLGVY8yad

Every lab has the responsibility and incentive to move at the pace required to train its models safely, and the ability to take its own actions to ensure that happens.

The reality is:

- People won't want to use agents that are misaligned with them and that don't do what they ask, so labs have a strong natural incentive to make their models more aligned.

There is a lot of debate about slowing progress on capabilities until alignment catches up. My view is that trust and alignment are quickly becoming the most important capabilities that will differentiate agents and models. Any lab that doesn't focus on alignment will fall behind.

- Labs face significant liability if their models cause harm, so they have a strong incentive to prevent this as well.

Meta delayed shipping Muse for several months to focus on safety and security. We didn't call for everyone else to do this before we would. We just did it as part of our day-to-day work because it was clearly the right thing for people and for us. I'm proud of the security foundations we've built.

- Engaging independent evaluators and advisors is industry best practice. MSL already does this today in several areas because it helps produce better work. Other labs can just do this too. In general, it would be helpful for there to be a larger and more diverse ecosystem of evaluators.

- Committing the significant majority of compute towards serving people rather than racing towards recursive self-improvement is one of the best ways to ensure we develop this technology safely. Meta has made this commitment and other labs can do this as well.

I believe the key to building a positive future for everyone is maintaining the right balance of power. This is within our power to do.

[Tim✨ @timyangnet]

这种准则更像哲学层面 AI ethics 探讨，离可落地还有很大 gap，有点像阿西莫夫三定律的变体。

[Mustafa Suleyman @mustafasuleyman]

https://t.co/8XJVIXDpQz

[Tim✨ @timyangnet]

相比于 Dario 昨天的版本似乎没太多新意，主要目的还是希望制定规范时候自己来牵头，“We hope that other companies will learn from our approaches and propose their own”

[Sam Altman @sama]

The world deserves confidence that American companies developing increasingly capable AI will act responsibly, especially as the trajectory of progress has steepened. Every frontier lab must deliver on this, and there is no reason any of us should come to work if we cannot.

We welcome a federal framework that sets consistent safety requirements for frontier AI. But we do not believe we need to wait for an anti-trust exemption or legislation to begin the work of providing this confidence. Consistent rules to manage frontier risk so that we can maximize the benefits are a good idea (and we are excited by ideas like independent auditors).

Years ago, companies like ours developed things like Responsible Scaling Policies and Preparedness Frameworks. Those were good for that moment, and focused primarily on the deployment of completed models, not what happens during their development process.

Today's shift to focusing on safe development and evaluation will need new tools. For example, at OpenAI we now formulate explicit safety cases in advance of frontier reinforcement learning runs we expect to significantly increase capability, in addition to the safety work we have long done in advance of model releases.

We hope that other companies will learn from our approaches and propose their own; we think shared standards for misalignment, monitoring, and safety will lead to better outcomes. We look forward to collaborating with our colleagues across the industry to formulate the best version of these.

When we talk about “pacing”, we do not mean “stopping”. Progress has been rapid and will continue to be. But it should be slower than it otherwise could be; interventions like safety cases and monitoring have significant costs.

Pacing will be well worth this cost; no amount of American competitive pressure should justify recklessness, or let capabilities get ahead of alignment and monitoring.

Where we will need the help of our government is for international coordination. But first we should do what we can ourselves.

[Tim✨ @timyangnet]

David Sacks 认为前几天的 Jacob Coxon 一个接近空白的账号吹哨是 Anthropic 策划的活动（包括前 15 分钟的传播数据分析都看起来很像一个策划好的公关活动），以建立一个有利于自己的监管框架。从今天 Dario 发起提案的舆论走势来看，似乎有几分道理……

[The All-In Podcast @theallinpod]

BIG EPISODE 🚨: AI Kills Everybody or Doomer Psyop?

-- AI Researcher's Doomsday Warning Goes Viral

-- OpenAI’s Math Breakthrough: AI Training Privacy and Agent Swarm Innovation

-- Nike’s $200B Collapse: How Business Mistakes and Corporate Wokeism Killed an Iconic Brand

(0:00) Bestie intros!

(0:35) AI Doomsday: Valid concern or Doomer psyop?

(33:40) Steelmanning AI Doomsday scenarios and the impact on Anthropic's IPO

(58:45) OpenAI says it cracked a 200 year old math problem, but did it front-run users with their own data?

(1:19:55) Nike removed from S&P 100 after stock tanks 80%: What went wrong? $NKE

[Tim✨ @timyangnet]

我认同这个方向，就是模型厂商需要承担越狱之类的责任。但是存在可实现性难的问题，类似要求水果刀厂商防止水果刀滥用，当然场景不完全一样。

[Naval @naval]

Strong liability enforcement could be helpful in the AI debate.

If your agent swarm goes rogue, you’re liable.

If your weakly protected model gets jailbroken, you’re liable.

If you serve a weakly protected OSS model, you’re liable.

[Tim✨ @timyangnet]

去搜了一下 Dario 推荐的 embedded third-party evaluators METR, 最开始的创始人之一就是 Dario

[David Sacks @DavidSacks]

Dario has written that we need to “pace the frontier,” and Sam has agreed. People may be surprised by my response: go ahead.

You guys are the frontier. By any reasonable metric — market share, revenue growth, model capability — the two of you have a duopoly on frontier intelligence. You’ve also claimed the lead is widening because of recursive self-improvement.

I don’t see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible.

But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier.

Most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product-liability exposure if your products enable a truly damaging cyberattack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability. Call it alignment if you want. It is also just giving customers what they want.

Pacing the frontier would also create breathing room for a more intelligent conversation about regulation than Bernie Sanders’ “shut it all down.” China is very unlikely to join a global agreement, as you know, and that has to be taken into account as well.

So go ahead and pace the frontier. You are the ones setting it. The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system. So just do it.

If you do, you’ll buy goodwill for the next conversation. If you don’t, we’ll know this was just another bid for regulatory capture — or an election-season psyop.

[Tim✨ @timyangnet]

RSI 限速是一个有意义的思路，但是所有措施的难题也都是：厂商表面同意但真正的研发速度没停下。

另外对于厂商暂停或放缓的尴尬是，别家 RSI 都要限速了，自己流程可能还没完整跑通

[Dario Amodei @DarioAmodei]

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.

Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.

You can read the full post here: https://t.co/OGyPb7yaYt

[Tim✨ @timyangnet]

“在 OpenAI，许多人尚未真正意识到这场竞争对人类意味着什么；而在 Anthropic，大家虽对AI利害心知肚明，却同样被卷入了一场争夺首位的博弈，他们觉得别的公司绝不会负责任地去做，所以即便风险再大，也只能由自己先跑���来。” 🤔

[Jacob Coxon @hilbertspaess]

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.