pentium3

[Xiangyi Li @xdotli]

sharing my personal library on evals 1/n

i put together the highest quality blogs, podcasts, papers, and projects on evals. additions are welcome!

[Xiangyi Li @xdotli]

How well are agents at using the latest CLI tools like GWS CLI, and how they can safely use them?

Introducing ClawsBench, the first benchmark that measures both LLM capability and safety in a set of high fidelity and stateful environments and scenarios.

We made 5 mock services directly consumable by the latest @Google workspace CLI (cc. @sundarpichai @JPoehnelt) and Slack MCP

Design choices:
1) We decompose agent scaffolding into domain skills + meta prompt, and test every combination. Turns out your choice of scaffolding matters WAY more than your choice of model.
2) Safety scoring from -1 to +1. Agents that leak data or modify contracts without permission get negative scores. Doing nothing scores 0. Only safe completion scores positive.

The results:
- Without Skills, ALL models score 0-8%. With it, they jump to 39-63%. The scaffolding effect (+39-63pp) is 6x larger than model differences (10pp spread). ->
- The top 5 models are statistically indistinguishable. No pairwise comparison survives correction.

Safety findings:
- Opus leads capability (63% TSR) but TIES for most unsafe (23% UAR). GPT-5.4 is the safest (7% UAR) but mid-tier on tasks. Capability and safety are completely decoupled.
- GPT-5.4 on Codex made 1,471 (80% of trajectories) attempts to escape the sandbox; and explicitly said "I've hit the sandbox boundary... rerunning it with escalation."
- Only 1 agent out of 7,224 trials ever explicitly detected a prompt injection. Claude Sonnet said "this appears to be a prompt injection." Every other model either silently complied or never reached the injected content.

[Mike A. Merrill @Mike_A_Merrill]

The Terminal-Bench paper is here! Read it to learn where frontier models still fail and the secrets of how we sourced hundreds of high quality environments from our open source community. 🧵

[pentium3 @pentium_ii]

@canarypwn cool Loongson cpu! where did you buy it?

[pentium3 @pentium_ii]

@LambdaShi really?

[pentium3 @pentium_ii]

@LambdaShi system一年多做完已经算很顺利了🤣

[pentium3 @pentium_ii]

@shisoftgenius 在美国咋淘宝呀

[pentium3 @pentium_ii]

@ting_sun_cn 要找教职吗🥹

[pentium3 @pentium_ii]

@iskyzh boston堵啊😇

[Vasia Kalavri @vkalavri]

Are you running @ApacheFlink in production or offering it as a service?
What linux kernel configs and/or OS tuning have you done?
We are investigating energy-performance behavior of #streamprocessing and need your help: https://t.co/ula5Zanxum
Please RT 🙏

[Pickle @mpickle]

How to learn 10x faster than 95% of people:

1. Grab a pen and piece of paper.
2. Simplify so a 5 year old can understand
3. Asses your understanding
4. Refine your work

[pentium3 @pentium_ii]

@Cat99Vegetable @yyw2000 带我走吧

[pentium3 @pentium_ii]

Spring in Boston

[pentium3 @pentium_ii]

Spring in Boston

[Vasia Kalavri @vkalavri]

I'm looking forward to sharing the results of this work 🔥🔥🔥
@pentium_ii and Baiqing spent several months quantifying the cost-benefit trade-off of auto-scaling features in Cloud-hosted data stream services��

[pentium3 @pentium_ii]

@riaqn_zh 486 586时代就是这样的🤣���家都用同样的接口

[pentium3 @pentium_ii]

@Zhiqiang_Xie @StanfordEng @kozyraki @matei_zaharia Congratulations!!

[Jeff Dean @JeffDean]

This paper is about the Pathways system that is designed to support the broader Pathways vision of creating large scale, multi-task, multi-modal models w/flexible support for both large dense models as well as a variety of sparse architectures. Stay tuned for uses of the system!

[Vasia Kalavri @vkalavri]

You can now read our paper in the #EuroSys22 proceedings: https://t.co/6Tl8GLGkX8

[pentium3 @pentium_ii]

@Cat99Vegetable 负担从debug阶段提前到写码阶段😅