Peter Schuller

Human. I'm KZY2KZZH on Threema, scode.01 on Signal.

[Peter Schuller @scode]

Opus and sol judges. I can try having it judge. But note that the actual task under evaluation here already includes extensive code review before it is even complete because I wanted to run the comparison on how I actually have agents build things in real life, which isn't just a one-off prompt. It's goal building with review of the goal followed by execution followed by review of the execution until convergence, etc.

[Peter Schuller @scode]

Apparently I need to adjust my skills or claude code will refuse to write sub agent reports to files even when explicitly told to.

[Peter Schuller @scode]

https://t.co/MUgwq32s9N

This is just one one test, but it's basically what I thought I'd see: With subscription subsidization it's cheaper to run sol (high) to complete the task compared to API prices for cheaper models. That's before taking into account that for actually doing this, I'd end up throwing sol/opus into the mix as an additional reviewer anyway.

If I were paying API prices for all models the story would be different.

(Those numbers are from having each model build a goal with up-front goad review on the plan, then code review as they go along during execution. So it's already showing the output after extensive adversarial review.)

I excluded Kimi 3.1 Pro because it's very expensive (it was multiple times more expensive than sol on some small tasks I did while getting the benchmarking harness done w/o showing signs of any amazing quality jump).

[Peter Schuller @scode]

Surprised so many model/harness combos have trouble waiting for a scripted event. claude code seems decent at it by default. Codex sucks - to the point that I have a whole skill just with a pre-written script and harsh instructions for the agent to not burn cached input tokens polling all the time (https://t.co/X1AvzuztnH). Grok bot also just randomly failed to succeed in having a script wake itself up.

[Peter Schuller @scode]

@NilsEngineer As I saw this, I'm literally setting up my own benchmark so it's cheaper for me to try new models. I'll be trying DeepSeek 4.1 and Kimi 3.1 at least. We shall see. :P

[Peter Schuller @scode]

I've actually changed my mind on this oldie but goodie. The principle I now prefer is that the stream flows in the direction of dependency which usually helps settle it. In this example that means (most of the time depending on context) that A is downstream of B which is the opposite of what I would have picked in 2015.

cc @bx

[Peter Schuller @scode]

Yeah public discourse is completely broken. On news, one recommendation is https://t.co/r8hSf4rL1P to read/listen/support to at least get rid of the constant bias in one way or another (they cover both sides, airing disagreements/different perspectives by staff etc).

I have no idea what to do. Elections/politics, news incentives, it's all broken in seemingly self re-enforcing ways. Both sides become deranged in different ways on different topics. It's very frustrating to watch constantly, I have to tune most of it out for sanity.

[Peter Schuller @scode]

If you've been hearing about the Starlink/Ukraine thing over the past week, you might be under the impression that what happened is that they slowed down internet speeds. Well, that's wrong.

I only realized because I accidentally stumbled upon reporting that wasn't misleading, after only hearing misleading reporting for days. I had a bunch of agents work on a TLDR of what actually happened and how it's been reported: https://t.co/x2dE0eWHEC

(I am making a point of quality of reporting, this is not an opinion on anything else.)

[Peter Schuller @scode]

@NilsEngineer I generally just use what's best overall. Given price per performance and speed I haven't really found anything that works competitively. I used GLM 5.3 flash for a while with a most expensive orchestrating agent, but didn't gain much on cost effectiveness and it's slow.

[Peter Schuller @scode]

@NilsEngineer Anthropic ToS is restricive. For OpenAI sure. I like Omp. :)

[Peter Schuller @scode]

If you're asking weather I think it matters in terms of the ethics of doing it, then not in a way that matters to me - it's a difference in degree but both shitty because the world needs to stop having countries invade other countries for territory out of nowhere.

I think it matters though including because:

- First off, just factual accuracy itself. Clearly misleading people is just one more reason to dismiss the media. This is just one example. Various things are regularly reported in wildly misleading ways.

- For anyone who, for whatever reason, doesn't think that it's shitty behavior to do this, it now becomes another source of polarization/media distrust. There is a legit ethical difference between "criple Ukrain's use of starlink relative to normal users" vs "revoke their special permission/ability to use it with high speed terminals, which is disallowed by default for everyone".

It's the same story as almost every political topic. On the off chance that someone on one side possibly pokes the head outside the echo chamber, they see legitimately ridiculous misleading reporting, therefore just confirming pre-existing biases about whatever the "other side" is perceived to be.

[Peter Schuller @scode]

Well, this is highly unscientific but better than "I used it for a while", and definitely not benchmaxxed: https://t.co/W5t1YECyZ9

Quality variance is basically noise at this sample size but still consistent with same ballpark performance - and way faster than both Luna and Haiku.

[Stefano Ermon @StefanoErmon]

1/4 Today, we’re releasing Mercury 3. Mercury 3 is by far our most powerful model and the most intelligent diffusion LLM ever released. It outperforms GPT-6 Luna and Haiku 5.5 on quality while running at 1000+ tok/sec.

[Peter Schuller @scode]

@samar_a_khanna It's now my goto for some markdown editing stuff, but will evaluate more for coding :)

[Peter Schuller @scode]

I just tried it, this is potentially exciting! I have only done basic stuff (repo navigation, listing PRs), but 2.5 fell immediately flat when I tried it. This worked fine, and it's *so so so* fast.

[Stefano Ermon @StefanoErmon]

1/4 Today, we’re releasing Mercury 3. Mercury 3 is by far our most powerful model and the most intelligent diffusion LLM ever released. It outperforms GPT-6 Luna and Haiku 5.5 on quality while running at 1000+ tok/sec.

[Peter Schuller @scode]

codex and claude code are so yesterday, let me introduce you to Palantír

[Peter Schuller @scode]

@sampullara I'm really starting to feel the climb towards this, in terms of the scope and complexity that passes the test for "not worth building something general, just vibe it up".

[Peter Schuller @scode]

Haha. I was daily driving a claude/codex load balancer I am building, and a bug caused a session to be unrecoverable. The other session which was debugging the problem instead hit a bug in codex-lb so now *that* session is unrecoverable.

(Luckily just asking an agent to read the other agent's transcript more or less works pretty well as a fallback resume mechanism.)

[Peter Schuller @scode]

Perfect example of agents still lacking judgement (in this case Opus 5.5).

Me: Hey lets turn a bunch of this stacked-PR handling skill into a deterministic script so we have less tool calls. I don't want to wait 30 seconds for the agent to churn tool calls for basic operations all the time.
Agent: Here we are, unit tests, e2e tests, and evals.
Me: Cool, merge.

Me in another session: Wtf is taking so long landing this stack? We disabled all github CI, this should be fast!
Agent: It's a fixed always-on 2 minute wait for every PR while waiting for a GitHub status check.

Me: *facepalm*

[Peter Schuller @scode]

Looks like I started exceeding the limit on 2026-09-12. I am fixing it. I am super curious how much this ends up explaining why Codex was so much less reliable than Claude, which I now wonder - maybe it was mostly in this repo (where I do various long unattended runs a *lot*).

It's muddled because I'm also having some *probably* unrelated issue where the model gets confused about how to spawn sub agents, though we'll see.

My confirmation biased brain remains more and more firm on "silent failure sucks".

[Peter Schuller @scode]

TIL that Codex *silently* (as far as I can tell) truncates project instructions injected at session start/post-compaction at project_doc_max_bytes bytes. And the default is 32kb.

[Peter Schuller @scode]

TIL that Codex *silently* (as far as I can tell) truncates project instructions injected at session start/post-compaction at project_doc_max_bytes bytes. And the default is 32kb.