Koutian Wu

PhD student @ UT Austin | My hamster called ASI | Agent/Eval | USTC alumnus | GeoAI & Earth modeling | ex-founder of a 1,300+ member student community

[Koutian Wu @ktwu01]

Every AI model has a scorecard now.

You’ve definitely seen it: GPT gets 85, Claude gets 87, Gemini gets 89...

And then suddenly everyone gets 90+.

Great. We solved intelligence.

Except we didn’t. The benchmark is just saturated.

So people keep building harder, more specialized, and frankly more ridiculous benchmarks to figure out what these models can actually do.

And now we have the opposite problem: there are so many benchmarks that nobody can keep track of the benchmarks anymore.

They’re scattered across arXiv, GitHub, Hugging Face, OpenReview, blogs, releases, everywhere.

So I built Benchmark Radar.

It continuously scans these sources, collects new benchmark-related work, and puts it into one searchable place.

We’ve already collected around 4,000 benchmark artifacts. You can see what came out today, what’s trending, search by topic, or export the whole dataset.

So instead of asking:

“What did GPT score on this benchmark?”

I’m asking:

“What benchmark are we all going to be arguing about next month?”

That’s Benchmark Radar.

https://t.co/EWPpVyqIvY

[Koutian Wu @ktwu01]

Benchmark Radar 使用案例 #2：写 benchmark-related post

假设你准备写一篇：

“最近有哪些新的 AI Agent benchmarks？”

问题是，benchmark 并不会只出现在一个地方。

有些先出现在 arXiv，有些先开 GitHub repo，有些出现在 Hugging Face，有些通过 release、RSS 或社区讨论传播。

如果每次写 post 都重新全网搜索，很容易漏掉长尾项目。

Benchmark Radar 的思路是：

先持续收集，再搜索。

我们每天从多个 source ingest benchmark-related artifacts。等你真正需要写 post 时，可以直接搜索过去积累的 corpus，而不是从零开始搜索互联网。

目前已经连续收集 20 天、累计数千条记录。

数据积累得越久，这种用法越有价值。

https://t.co/ldlRicDjLr
https://t.co/8wQv1LBY5D

[Koutian Wu @ktwu01]

who is him

[Koutian Wu @ktwu01]

this guy @ACongroo working despite his favorite dish lol

[Koutian Wu @ktwu01]

ChatGPT 居然开始搞防沉迷了，家长可以设置对孩子的限制。
#chatgpt

[Koutian Wu @ktwu01]

Benchmark Radar 使用案例 #1：写论文时找 Related Work

你正在写一篇论文，需要回答一个很基础但很耗时间的问题：

这个问题以前有没有人做过？有哪些 benchmark 和我的工作最接近？

通常你可能要分别去 Google Scholar、arXiv、GitHub、Hugging Face、OpenReview 搜关键词，再自己判断哪些是真的 benchmark。

现在可以先直接在 Benchmark Radar 里搜。

我们持续从多个 benchmark-related sources 收集信息，把分散在论文、GitHub repo、Hugging Face、release 和社区讨论里的 benchmark 聚合到一起。

例如你可以直接搜：

scientific agent benchmark
climate benchmark
code repair benchmark
multimodal reasoning
Earth science agent

先快速建立 benchmark landscape，再决定哪些工作值得深入读。

这也是我们做“一键导出”的原因：Benchmark Radar 不只是日报，也可以逐渐变成一个 benchmark research corpus。

https://t.co/8wQv1LBY5D
https://t.co/ldlRicDjLr

[Koutian Wu @ktwu01]

我从 GitHub、Hugging Face、OpenAlex 等 11 个来源抓了 4000+ 条 benchmark/eval/dataset 数据，如果需要寻找 related work 或找到适合eval自己的agent 的 bench 或者关注最新的 eval 进展，可以看这里：https://t.co/EdZboa4k0p ，每天更新，并支持一键导出数据

[Koutian Wu @ktwu01]

Even after 17 years, I still see this as very insightful.

[Koutian Wu @ktwu01]

This article by @paulg 17 yrs ago is for every founder, founding team, startup, and everyone who wants to become a founder.
Really insightful: don't be hapless. Avoid being passive and trying to solve every outside obstacle you meet, trying to learn from successful startups, trying to do things That solves your problems.

Let's review the last sentence of this article. "Make something people want" is the destination, but "be relentlessly resourceful" is how you get there.

https://t.co/xEqBNZltwI

[Koutian Wu @ktwu01]

wtf'？

[DeMo❤️‍🔥 @0xDeMoo]

顶级牛逼, 提前10年预知未来

kimi创始人杨植麟, 2015年写的歌

当年的苦逼穷学生, 如今歌里唱的都有了

兰博基尼、美女、钞票、改变世界

[Koutian Wu @ktwu01]

太搞笑了！Claude 把自己以为是王者荣耀了
直接开启防沉迷模式

[Koutian Wu @ktwu01]

Benchmark Radar code 更新：

新增一键导出：方便直接搜索和分析我们持续收集的 benchmark corpus。目前已经连续收集 20 天，累计收录 3,995 个例子。以后写 benchmark-related post、做 benchmark landscape analysis，或者寻找某个领域已有 benchmark 时，可以先直接在这里搜索，而不需要再去全网多个平台逐个检索。随着数据持续积累，这个 corpus 会覆盖更多长尾 benchmark。
新增 RSS 订阅：现在可以直接通过 RSS 阅读每天的 Benchmark Radar 日报，不需要每天手动打开网站查看。
扩展数据源：在原有 GitHub、Hugging Face、arXiv、OpenAlex 等 API/RSS 数据源基础上，新增 Alina 的信源。目前 pipeline 共包含 11 个 sources，覆盖论文、代码仓库、模型/数据平台、release、first-party feeds 和 Hacker News 等不同类型的信号。

目前一次 ingest 的结果包括：

GitHub：300+
OpenAlex：237
Hugging Face：85
arXiv：37
Hacker News：13
GitHub Releases：4
First-party feeds：1

https://t.co/ldlRicCLVT
https://t.co/8wQv1LBqg5

[Koutian Wu @ktwu01]

没错，我就是把它当成精神鸦片来用的

[Koutian Wu @ktwu01]

只有能够让人沉迷的东西才配防沉迷，对吧？说明大家已经公认Claude Code这玩意是非常沉迷的精神鸦片、精神毒品 hhhhhh

[Koutian Wu @ktwu01]

只有能够让人沉迷的东西才配防沉迷，对吧？说明大家已经公认Claude Code这玩意是非常沉迷的精神鸦片、精神毒品 hhhhhh

[Koutian Wu @ktwu01]

Anthropic has more confidence in my financial future than I do.

Spent this month: $10.48
Default monthly spend limit: $200,000
Bro thinks I’m training Claude 6 in my basement.

Seriously though, check your billing limit:
https://t.co/42ZAKSmM7A

[Koutian Wu @ktwu01]

⚠️ 小心点，不要中招 Anthropic……

Anthropic apparently defaults to assuming every Claude API user is a $200k/month customer.

我的账号：$10.48 spent
Anthropic：Monthly spend limit: $200,000
Thanks for believing in me, Anthropic 😭

Seriously though, check your billing limit:
https://t.co/ybXQH9KylC

[Koutian Wu @ktwu01]

小心点，不要中招 Anthropic.... WARNING: If you are using #claude platform. Anthropic default thinking that you are a 200k monthly USD user for Claude.
这个 Claude default setting 太搞笑了
https://t.co/G72xt6xpbD
Just don't—never use the default setting. Otherwise, you will pay $200,000 until you get notified.

[Koutian Wu @ktwu01]

Used up all the Claude Code credits provided by UT Austin. Actually, I think this is very generous. The Claude Code credits—about 100 US dollars expiring in two to three months—I used ahead of time.

#utaustin #claudecode #ai