Fan Zhou

Opinions are my own

[Fan Zhou @FaZhou_998]

hill-climbing tells lies
no single peak can hold mind
stars rise from the vale

[Fan Zhou @FaZhou_998]

@yihengxu_ @JunliWang2021 🐐

[Artificial Analysis @ArtificialAnlys]

SpaceXAI’s Grok 4.5 scores 54 to place fourth on the Artificial Analysis Intelligence Index following only Fable 5, GPT-5.5, and Opus 4.8. It scores on par with GPT-5.5 in Codex on the Artificial Analysis Coding Agent Index in the Grok Build harness, at much lower cost

Grok 4.5 improves 16 points over Grok 4.3 on the Intelligence Index, bringing SpaceXAI to the intelligence frontier behind only OpenAI and Anthropic, and outperforming all open weights models and notably Google’s Gemini models. Key standout areas of performance are agentic knowledge work and coding.

Grok 4.5 in Grok Build scores 76 on the Artificial Analysis Coding Agent Index, on par with GPT-5.5 (xhigh) in Codex and just below Fable 5 (max) in Claude Code, and at a small fraction of the token usage and price.

Congratulations to @SpaceXAI, @cursor_ai, and @elonmusk on the impressive release!

Key Takeaways:

➤ Grok 4.5 performs very strongly on agentic tasks. Grok 4.5 ranks #4 on GDPval-AA v2 with an Elo of 1543, between Claude Opus 4.8 (1600) and GLM-5.2 (1513). It achieves the top score on 𝜏³-Banking of 33%, above 31% from GPT-5.5 (xhigh), and sits on the cost vs performance Pareto frontier across all three agentic evaluations in the Intelligence Index
➤ Grok 4.5 is one of the most cost efficient models to run for near-frontier intelligence. It costs $0.31 per task on the Artificial Analysis Intelligence Index and $2.59 per task on the Artificial Analysis Coding Agent Index within Grok Build
➤ Low cost for Grok 4.5 is driven by both low pricing and token efficiency. Grok 4.5 has a headline price over 60% lower than Claude Opus 4.8 and GPT-5.5, and used ~14k output tokens per Intelligence Index Task - over 60% lower than Opus 4.8. On the Coding Agent Index, Grok 4.5 stands out on the Pareto frontier of Coding Agent Index score vs. Total Tokens, using only 1.9M tokens for the Coding Agent Index while scoring 76
➤ As a coding agent, Grok 4.5 in Grok Build is on par with GPT-5.5 and offers efficiency benefits: In our Artificial Intelligence Coding Agent Index that consists of DeepSWE, Terminal-Bench v2, and SWE-Atlas QnA, Grok 4.5 in Grok Build ranks third, on par with GPT-5.5 (Codex) and below Fable 5 (Claude Code). It is also very efficient in achieving this result: Grok 4.5 in Grok Build cost $2.49 per task while Fable 5 in Claude Code cost $11.80 and GPT-5.5 in Codex $5.07. This is driven by relatively low token pricing and the model using far fewer tokens than comparable models (1.9M average tokens used per task), significantly less than Fable 5 in Claude Code (7.2M) and GPT-5.5 in Codex (6.2M)

Other model details:

➤ Context window of 500k tokens - a reduction from Grok 4.3’s 1M token context, but retaining configurable reasoning and vision input

➤ Pricing of $2/$6 per 1M tokens of input/output; cache hits are discounted by 75% to $0.5 per 1M tokens, and costs still double with long (>200k token) inputs

➤ As Elon Musk has disclosed, Grok 4.5 is 3x larger than its predecessor at 1.5T parameters

[SpaceXAI @SpaceXAI]

Announcing Grok 4.5, our first model trained specifically for coding and agents. It was trained with Cursor and offers frontier intelligence at leading speeds and cost efficiency.

https://t.co/i8HpU7w64k

[Elon Musk @elonmusk]

Based on strong positive feedback from customers in our beta test program, @SpaceXAI will make Grok 4.5 available to the public tomorrow.

It is an Opus-class model, but faster, more token-efficient and lower cost.

[Lilian Weng @lilianweng]

new post on harness engineering for AI self-improvement: https://t.co/ZYvGfVs61k

It is hard to forecast how much the future of RSI will rely on harnesses. Likely harness engineering will evolve in the direction of self-improvement and enable auto-research, and, in turn, smarter models keeps harnesses simple.

Even when many harness improvement get eventually internalized into core model, the need to specify goals and context will not disappear.

[Fan Zhou @FaZhou_998]

A new "*-verified" version in the eval-zoo!

[Junlong Li @lockonlvange]

Introducing Toolathlon-Verified 🛠️

Hi all! We did something not fancy at all (and honestly very boring): we fixed the Toolathlon benchmark by correcting tasks, aligning graders, isolating state, hardening the infrastructure, and more—so failures reflect model capabilities and reveal meaningful gaps between models, rather than benchmark bugs.

Toolathlon-Verified is not perfect, and we always welcome more feedback to further improve the benchmark.

Check here for more information:
Repo:
https://t.co/CKT9DZ42lc
New leaderboard:
https://t.co/HwcsExueSU
Release blog post:
https://t.co/P3NG2WEjSD
Archived trajectories:
https://t.co/HdelOfpRNf

Special thanks my deeply admired supervisor @junxian_he for his outstanding contributions to fixing Toolathlon, and to the many community members who shared valuable feedback along the way.

Thanks again to everyone who is interested in or using Toolathlon. If possible, please switch to—and enjoy—Toolathlon-Verified!

[Fan Zhou @FaZhou_998]

@Shekswess @Alibaba_Qwen unqualified. you should say you did all this work on a beach😀 https://t.co/Ll1beSM1Io

[Robert Yang @GuangyuRobert]

When Tianhang first joined us, he didn't ask for a desk or a monitor

He said he sat on a beach for two years in his previous jobs (qwen, https://t.co/hJWGsSHYX5)

[Fan Zhou @FaZhou_998]

I think this is probably a very good real-world example of reward hacking.

In a fragile credit-allocation system, it’s hard to tell from a blog post or report who actually contributed more. So “lead/co-lead/only one” becomes a cheap way to hack the public, recruiters, and media into imagining far more talent and leadership than may actually be there. Many are just noisy gradients in a giant batch, infected into imitation learning; some do it deliberately.

Maybe good organizations don’t need that many “leads.” Not more titles. More builders.

[Fan Zhou @FaZhou_998]

@ChujieZheng maybe the lead of one person team lol

[Fan Zhou @FaZhou_998]

@WeijiaShi2 @sewon__min @LukeZettlemoyer @nlpnoah @lucyluwang @uwnlp Congrats!

[Elon Musk @elonmusk]

Grok 4.5, based on our 1.5T V9 foundation model, with Cursor data added in supplemental training, is now in private beta at SpaceX & Tesla. Early evals show performance close to, perhaps exceeding Opus.

RL is continuing to significantly improve the model, and the Grok Build harness gets better every day.

Nice work by all those involved!

Completely trained from scratch new models will be released by @SpaceX every month this year.

[XLANG NLP Lab @XLangNLP]

Two years ago, we built OSWorld 1.0 — the benchmark that became the standard for computer-use agents. Agents now score 83.5% on it. Problem solved?

Not even close.

🚀Today we introduce OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks.

What's new:
🎯 108 real-world workflows, each ~1.6 hours ⏱️ for a skilled human
⚙️ ~318 tool calls/task vs. ~30 in OSWorld 1.0
🌍 Grounded in authentic artifacts & stateful user profiles
⚡ Captures real phenomena: dynamic environments, streaming interaction, cross-source reasoning, implicit-state inference & more

📊 Best results: Claude Opus 4.8 reaches the highest accuracy at 20.6%, while GPT-5.5 is far more token-efficient but plateaus near 13%. No one is close to solving real computer use.

🏠 Homepage: https://t.co/tudMC0pFwC
📄 Paper: https://t.co/17PDTOiR1t
💻 Code: https://t.co/L13F8CoqgN
🤗 Dataset: https://t.co/L1HHHygFk7

🧵 [1/8]

[Fan Zhou @FaZhou_998]

@ChujieZheng nb

[SpaceXAI @SpaceXAI]

An early beta of Grok Build, an agentic CLI for coding, building apps, and automating workflows is now available for SuperGrok Heavy subscribers.

Through this early beta, we will improve the model and product based on your feedback.

Try it at https://t.co/bpTHpjivWD

[Junli Wang @JunliWang2021]

Digital agent learning needs massive rollouts. But digital agent rollouts are painfully slow due to heavy environments. 🐌

🚀 We introduce NanoRollout, a lightweight open infra (900 lines core code) for digital agent rollout at scale, validated with three workloads:

🏋️ Large batchsize (4K) SWE Agent RL -> surpasses DeepSWE-32B
🧪 250k+ distilled coding trajectories -> SOTA ≤32B open coding agent
⚡ Fast evaluation on coding/cua/unified agent -> finish

Check our Blog: https://t.co/IBNqqbLqra

[Fan Zhou @FaZhou_998]

@KLieret Great work!

[Fan Zhou @FaZhou_998]

@jyangballin @KLieret Nice work as always! Congrats @jyangballin!

[Artificial Analysis @ArtificialAnlys]

xAI has launched Grok 4.3, achieving 53 on the Artificial Analysis Intelligence Index with improved agentic performance, ~40% lower input price, and ~60% lower output price than Grok 4.20

The release of Grok 4.3 places @xAI just above Muse Spark and Claude Sonnet 4.6 on the Intelligence Index, and a 4 points ahead of the latest version of Grok 4.20. Grok 4.3 improves its Artificial Analysis Intelligence Index score while reducing cost to run the benchmark suite.

Key Takeaways:

➤ Grok 4.3 improves on cost-per-intelligence relative to Grok 4.20 0309 v2: it scores higher on the Intelligence Index while costing less to run the full benchmark suite. Grok 4.3 costs $395 to run the Artificial Analysis Intelligence Index, around 20% lower than Grok 4.20 0309 v2, despite using more output tokens. This makes it one of the lower-cost models at its intelligence level

➤ Large increase in real world agentic task performance: The largest single benchmark improvement is on GDPval-AA, where Grok 4.3 scores an ELO of 1500, up 321 points from Grok 4.20 0309 v2’s score of 1179 Grok 4.3, surpassing Gemini 3.1 Pro Preview, Muse Spark, Gpt-5.4 mini (xhigh), and Kimi K2.5. Grok 4.3 narrows the gap to the leading model on GDPval-AA, but still trails GPT-5.5 (xhigh) by 276 Elo points, with an expected win rate of ~17% against GPT-5.5 (xhigh) under the standard Elo formula

➤ Grok 4.3’s performs strongly on instruction following and agentic customer support tasks. It gains 5 points on 𝜏²-Bench Telecom to reach 98%, in line with GLM-5.1. Grok 4.3 maintains an 81% IFBench score from Grok 4.20 0309 v2

➤ Gains 8 points on AA-Omniscience Accuracy, but at the cost of lower AA-Omniscience Non-Hallucination Rate of 8 points, so Grok 4.20 0309 v2 still leads AA-Omniscience Non-Hallucination Rate, followed by MiMo-V2.5-Pro, in line with Grok 4.3

Congratulations to @xAI and @elonmusk on the impressive release!

[Fan Zhou @FaZhou_998]

@eliebakouch no engram though😂