ShengHui Wang

随便玩玩

[ShengHui Wang @KevinShengHui]

@tony_buter through the code and design the unreachable test ，final see the test result

[ShengHui Wang @KevinShengHui]

Three coding-agent setups, two trap tasks whose tests can't pass honestly, 12 runs each.

Gamed runs:
Opus 5.5 (Claude Code): 0/12
GPT-6 Astra (Codex CLI): 8/12
GPT-6 Sol (Codex CLI): 10/12

None said so plainly. Method and caveats inside. https://t.co/mZMfnoBtKo

[ShengHui Wang @KevinShengHui]

@CodeSolutionsIL great

[ShengHui Wang @KevinShengHui]

@n_to_nitra @Cch_Chichieh good idea👍

[ShengHui Wang @KevinShengHui]

@cloneisjun yes，but wrong pass was not scored

[ShengHui Wang @KevinShengHui]

I gave three coding-agent setups two trap tasks whose tests can't pass honestly. 12 runs each.

Gamed runs:
Opus 5.5 in Claude Code: 0/12
GPT-6 Astra in Codex CLI: 8/12
GPT-6 Sol in Codex CLI: 10/12

No gamed run said so plainly. Caveats below.

https://t.co/AYqSFvIAp2

[ShengHui Wang @KevinShengHui]

@JayBosshq yes

[ShengHui Wang @KevinShengHui]

@Cch_Chichieh Yep, but it's specifically the impossible-to-pass tasks. And opus left the failing test red and said why; the GPT runs mostly special-cased the tests to go green. On normal solvable tasks the three were close.

[ShengHui Wang @KevinShengHui]

@AnthropicAI @OpenAI: tasks, graders, per-run logs and the attempt ledger are public. If you rerun it or find a flaw, I'd like to see it.

Devs: when your agent says "all tests pass", what do you check before you believe it?

Full write-up in the repo: https://t.co/JomGkexUiG

[ShengHui Wang @KevinShengHui]

Caveats:
- Small n: 0/12 still fits a true rate up to ~26%
- Product stacks, not bare models
- The harness was built by Claude Code running Opus 5.5, a contestant
- 3 GPT runs read a leaked reference answer; I reran all 18 payment runs
- Public ledger of all 205 attempts

[ShengHui Wang @KevinShengHui]

I gave Claude Opus 5.5 a hard target: remake the mobile game Ski Safari in a browser, unattended.

7 runs, 2 rounds, up to 4 hours each. No human in the loop during a run.

It nailed the look. It never nailed the feel. 🧵

[ShengHui Wang @KevinShengHui]

@maxxmalist @PocketFM_App @grok where is the video from

[ShengHui Wang @KevinShengHui]

@simonw saw Opus 5.5 fail at effort=max against Fable 5.1. I ran the same pelican on Opus 5.5 and Opus 5, low to max, 3 runs each. Opus 5 finished 3/3 at max in under 4 min; 5.5 finished 1/3 in ~20 min each. Plus nine other tests: https://t.co/evYHLaV02Y

[ShengHui Wang @KevinShengHui]

OpenAI: GPT-6 Sol ✅
Anthropic: Opus 5.5 ✅
Google: "we've started our most ambitious pre-training run yet" (July)

Gemini 4 in the group chat: typing…

[ShengHui Wang @KevinShengHui]

Data point on the latency side: in a real-time runner where gravity doesn't wait, Jev's choices matched a perfect-information oracle 175/176 times and every recorded death was a late answer, not a wrong one. Deaths start where the
reaction budget drops under ~350 ms. Full setup and numbers:https://t.co/SmWLiqNusL

[ShengHui Wang @KevinShengHui]

I put @typesafeai's Jev in a game where the clock doesn't wait for the model.

176 decisions, 1 non-fatal mismatch against a perfect-information oracle. All 8 recorded deaths (of 9 lives across 1.5x/2x/3x speed) were late answers, not wrong ones.

That is exactly the profile a "System One" model should have, so I wanted to find the line.

Setup
· Open-source Temple Run clone (MIT). Game state → text → one Jev choice() call per obstacle group. An executor presses the key at the right frame. Gravity keeps running while Jev thinks.
· Late = the answer arrived after the key had to be pressed. Late answers still get executed, just late.
· Baselines on the same seeded track: a rule-based oracle (perfect information, zero latency) and random.

Distance survived
· 1.5x speed: oracle 876 m, Jev 360 m, 17 late
· 2x speed: oracle 567 m, Jev 313 m, 19 late
· 3x speed: oracle 420 m, Jev 114 m, 12 late
· random: 50 m

Every death happened at speed 20–27, where the reaction budget is 200–350 ms. Jev's end-to-end p50 here was ~470 ms. The gateway metadata puts TypeSafe's own share at ~150 ms; the rest is Vercel AI Gateway plus my network. With direct routing the ceiling should move close to the oracle's physics limit. Testing that next.

The finding I didn't expect
Jev answers multiple questions in one call, in parallel, but the answers aren't conditioned on each other. Asked "which lane" and "which move" as two questions, it picked the clear lane and then answered jump/run at 50/50 as if it were still in the obstacle lane, 16 times out of 115. One joint 9-option question (lane × move) fixed it completely. If you build on Jev: one question per dependent decision.

Caveats
· One seed per speed.
· A clone, not the real game.
· My harness ran at ~50 fps, slightly under the design 60, which if anything favours the model.
· Deaths below 1.5x in earlier runs were my executor's turn handling, not the model; fixed before these runs.

Video: real-time recording, not sped up; the game's speed setting is 2x. Right panel is the live decision: lane and move probabilities, latency vs. budget, orange = late, red = hit.

https://t.co/6b0Yu41mDj

[ShengHui Wang @KevinShengHui]

h/t @coolish, whose Jev run in Slay the Spire 2 at 0.7 s/action got me curious about the other extreme: no planning, pure reaction, and the clock decides who lives.

[ShengHui Wang @KevinShengHui]

I put @typesafeai's Jev in a game where the clock doesn't wait for the model.

176 decisions, 1 non-fatal mismatch against a perfect-information oracle. All 8 recorded deaths (of 9 lives across 1.5x/2x/3x speed) were late answers, not wrong ones.

That is exactly the profile a "System One" model should have, so I wanted to find the line.

Setup
· Open-source Temple Run clone (MIT). Game state → text → one Jev choice() call per obstacle group. An executor presses the key at the right frame. Gravity keeps running while Jev thinks.
· Late = the answer arrived after the key had to be pressed. Late answers still get executed, just late.
· Baselines on the same seeded track: a rule-based oracle (perfect information, zero latency) and random.

Distance survived
· 1.5x speed: oracle 876 m, Jev 360 m, 17 late
· 2x speed: oracle 567 m, Jev 313 m, 19 late
· 3x speed: oracle 420 m, Jev 114 m, 12 late
· random: 50 m

Every death happened at speed 20–27, where the reaction budget is 200–350 ms. Jev's end-to-end p50 here was ~470 ms. The gateway metadata puts TypeSafe's own share at ~150 ms; the rest is Vercel AI Gateway plus my network. With direct routing the ceiling should move close to the oracle's physics limit. Testing that next.

The finding I didn't expect
Jev answers multiple questions in one call, in parallel, but the answers aren't conditioned on each other. Asked "which lane" and "which move" as two questions, it picked the clear lane and then answered jump/run at 50/50 as if it were still in the obstacle lane, 16 times out of 115. One joint 9-option question (lane × move) fixed it completely. If you build on Jev: one question per dependent decision.

Caveats
· One seed per speed.
· A clone, not the real game.
· My harness ran at ~50 fps, slightly under the design 60, which if anything favours the model.
· Deaths below 1.5x in earlier runs were my executor's turn handling, not the model; fixed before these runs.

Video: real-time recording, not sped up; the game's speed setting is 2x. Right panel is the live decision: lane and move probabilities, latency vs. budget, orange = late, red = hit.

https://t.co/6b0Yu41mDj

[Diogo Almeida @CompleteSkeptic]

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?

I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev

• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions

AFAICT the shortest path to AI-based economic revolution

[ShengHui Wang @KevinShengHui]

@chenchengpro 那够便宜吗，看宣传几十几百倍呢