Josselin Perrus

Product Manager

[Josselin Perrus @nonils]

@raw_works @threepointone @aidenybai I'd very much like to understand more specifically how prose us used to match @threepointone question. Do you have example / blog? Genuinely curious

[Thorsten Ball @thorstenball]

https://t.co/PV4OLiTZnu

[Josselin Perrus @nonils]

@ryancarson @joshferrara I find creating the acceptance criteria the critical but tedious part. Do you have tips to help with that?

[Josselin Perrus @nonils]

spot on: there a is a Skill issue, and there is the skill issue. I realized that a Skill could only do so much for me, I just needed to get better at planning at technical level.

[Matt Pocock @mattpocockuk]

Things people get wrong with my grill-* skills:

- Being too passive
- Not grilling in parallel
- Not prototyping
- Going into the dumb zone
- Grilling too hard
- Grilling too large a topic
- Using too dumb a model
- Clearing the context too soon

Here's the breakdown:

[Josselin Perrus @nonils]

@Scaleway_fr I actually could not do it: kept erroring

[Josselin Perrus @nonils]

OMG @Scaleway_fr your dedibox onboarding flow is so ***

[Josselin Perrus @nonils]

@ryancarson I see you have properly expressed Acceptance Criteria. 2 questions:
- Did you generate them automatically or manually?
- Are they part of a spec? How are they inputed to Codex?

[Josselin Perrus @nonils]

@elvissun Codex is the « harness », isn’t it? Would you elaborate on what exactly you gave Codex that made a difference ? How did you implement the feedback / back pressure?

[Manuel Faysse @ManuelFaysse]

I feel what these results (@mattjustram) mainly show is NOT that retrievers like BM25 are better than previously thought, NOT that GPT 5.5 is amazing at search orchestration, but rather that scaffolds matter a lot in search (and should be reported).

Pi-serini is one of the first non-trivial search specific efforts to give some proper tooling to LLMs (cached results, well defined tool calls) and smashes the baseline with the same retriever and baseline model. @antoine_chaffin 's results with/without the "get_document" tool also vastly differ: purely scaffolding. CC and Codex are good scaffolds by design and also perform very well at search as shown here (although they are not built specifically for this).

What does this mean:
Biggest low hanging fruits right now in search probably aren't through better embedding models (or even better baseline LLMs) but smart scaffolding paradigms (think RLMs @a1zhang, Pi-serini @lintool, Deep Research systems @din0s_ ).
As LLMs get better, I would expect inductive biases cooked in the scaffolds to become less important than now. However, scaffolds will still need to be expressive enough to let the baseline model "cook" and still need to encode what matters to the end user (speed matters a lot, search is not just accuracy).

Importantly, I believe scaffolds are not an inference time thing only, but should serve during document indexing. To properly contextualize documents in a corpus, we need to move away from the "1 paragraph/webpage is 1 document" concept and use agents to properly map the corpus space, and leverage their priors to index then retrieve. Test time compute scaling is not reserved to querying agents!

Just a few thoughts on recent results, this direction will be well aided by novel datasets like @dianetc_ 's Obliq-Bench which will necessarily require a change in paradigms, the easiest surely being around scaffolding.

[Josselin Perrus @nonils]

@mattpocockuk as non-dev, having some trouble picturing the LOGIC.md from your prototype skill. Would you have concrete examples?

[Josselin Perrus @nonils]

so /goal translates your intent into a verifiable stop condition and iterates towards verifying that condition

but translating intent into verifiable conditions is the hard/ambiguous part, isn't it?

[Dan McAteer @daniel_mac8]

this is amazing. do this.

get codex to write its own /goal

/goal + gpt-5.5 high + fast mode is the highest leverage ai agent configuration available today

[Josselin Perrus @nonils]

@Teknium @Saboo_Shubham_ do you get to see how Hermes translates instructions into acceptance criteria?

[Josselin Perrus @nonils]

@diggz976 @GaryMarcus if you may share: you talk about equipments / manual / repair guides, what kind of equipment are we talking about? Factory or something like that? It sounds like a cool use case, but too many blanks for my limited imagination

[Josselin Perrus @nonils]

@charlieholtz I can't rename threads. I'm on 0.52.1 on mac. Unless "Rename directory" means something else

[mohamed @momito]

i’ve been playing around with flue, and have been really enjoying it

i’ve created a visual guide, let me know what you think

https://t.co/eU0vn8aNg7

[Dilum Sanjaya @DilumSanjaya]

Fun interactive science app ideas | Part 3

Played around with generating 3D biological structures and made an app to explore them interactively

UI Design
GPT Images 2

Code
Gemini 3.1 Pro

More demos ↓

[Josselin Perrus @nonils]

@diggz976 @GaryMarcus Would you share in more details?

[Ahmad Awais @MrAhmadAwais]

how did we make deepseek outperform opus 4.7?

i've been thinking about why "open model bad at tool calling" is almost always a harness problem, not a model problem.

context: spent the two days looking at billions of tokens in @CommandCodeAI (tb open source ai cli) using deepseek. I ended up writing a tool-input repair layer. the trigger was watching deepseek-flash fail on the simplest /review run, every shellCommand and readFile call bouncing back with a raw zod issues blob, the model unable to recover because the error wasn't in a form it could read. by the end deepseek v4 pro was beating opus 4.7 6/10 times on our internal evals.

a few things i learned that feel general:

1/ the failure modes aren't random they're a small finite compositional set.

across deepseek-flash, deepseek v4 pro, glm, qwen, the same four mistakes repeat almost exactly:
- sending `null` for an optional field instead of omitting it
- emitting `["a","b"]` as a json *string* instead of an actual array
- wrapping a single arg in `{}` where the schema expected an array (an "empty placeholder")
- passing a bare string where an array was expected (`"foo"` instead of `["foo"]`)

four repairs, ~30-100 lines each, ordered carefully (json-array-parse must run before bare-string-wrap or `'["a","b"]'` becomes `['["a","b"]']`). that is the whole catalogue. when i hear "this open source model can't do tool calls" i now assume one of those four, and so far that's been right ~90% of the time.

2/ the funniest failure mode is also the most revealing.

deepseek-flash, when asked to edit or write a file, sometimes emits the path as a *markdown auto-link*:

filePath: "/Users/x/proj/[notes.md](http://notes. md)"

our writeFile tool obediently trued creating files literally named `[notes.md](http://notes .md)` until we caught it. this is not a hallucination. it's the post-training chat distribution leaking through the tool boundary the model has been rewarded for auto-linking in conversational output, and is applying that prior in a context where it makes no sense. the fix is two regex lines that unwrap only the degenerate case where link text equals url-without-protocol real markdown like `[click](https://x .com)` passes through untouched.

this is also conditioning of their own tools during RL which were different from all other tools we write and ofc can't predict.

"tool confusion" is a more useful frame than "capability gap." the model knows how to format a path. it just hasn't been told clearly enough that this path is going to fopen, not into a chat bubble. so we encode that hint at the schema level `pathString()` instead of `z.string()` and the leak is plugged for every path field at once.

3/ the design choice that mattered was inverting preprocess-then-validate to validate-then-repair.

my first attempt was the obvious one: a preprocessing pass that normalized inputs (strip nulls, parse stringified arrays, etc.) before zod ever saw them. it broke immediately, writeFile content that *happened* to be json-shaped got rewritten before it hit disk. silent corruption, easy to miss in a smoke test.

then i made it less greedy
- parse the input as-is. if it succeeds, ship it. valid inputs are never touched.
- on failure, walk the validator's own issue list. for each issue path, try the four repairs in order until one applies.
- parse again. on success, log `tool_input_repaired:${toolName}`. on failure, log `tool_input_invalid:${toolName}` and return a model-readable retry message.

the structural insight here is: when you preprocess, you encode a prior about what's broken. when you let the validator complain first, the schema is the prior, and you only spend repair budget at the exact paths the schema actually disagreed at. the validator is doing the work of localizing the bug for you. it's the same shape as cheap-then-careful everywhere else try the fast path, fall back on evidence.

(this also gives you per-tool telemetry for free. you can watch repair rates per (model, tool) and notice when a model regresses on a specific contract before users do.)

4/ shape invariants and relational invariants need different fixes.

the four repairs above all handle shape problems wrong type, missing key, wrong container. but read_file had a *relational* invariant: "if you provide offset, you must also provide limit, and vice versa." deepseek kept calling `readFile({ absolutePath, limit: 30 })` and getting an `ERROR:` back. you can't fix this with input repair, because each field is independently valid the bug is in the relationship between them.

so i taught the function the model's intent instead. `limit` alone → `offset = 0`. `offset` alone → `limit = 2000` (matches common read tool ops default). then surfaced the decision back to the model in the result:

"Note: limit was not provided; defaulted to 2000 lines. To read more or fewer lines, retry with both offset and limit."

no `Error:` prefix, so the tui doesn't paint it red. the model sees what we picked and can self-correct on the next turn if our guess was wrong. transparency over silent magic wins big.

repair where you can. extend semantics where you can't. surface the choice either way.

zoom out:

a lot of what looks like model capability is actually contract design. a strict schema is a choice with a cost it filters out noise, but it also filters out recoverable noise from any model that hasn't memorized the exact json contract you happened to pick. the largest commercial models eat that cost invisibly and are linient on tool calling because they've seen enough of every contract during pretraining; open models pay it loudly and get dismissed for it.

the harness is where you mediate between distributions. four small repairs (i'm sure more to follow as we have three more merging today), two regex lines for auto-links, one relational default, one prefix change. the model didn't change. the contract got more forgiving in exactly the places it needed to be.

deepseek v4 pro now beats opus 4.7 6/10 times on our internal evals.

imo "skill issue" applies to the harness more often than the model.

[Josselin Perrus @nonils]

To illustrate what I meant with the auth thing: I had codex track down all the token / keys and their location. Maybe a skill issue on my end, but the mental model is hard to come by.

[Josselin Perrus @nonils]

@pydantic Logfire folks, Codex is having a hard time working with your CLI/MCP. Latest hicup, Codex errored itself by running a CLI command that installed a local MCP that blocked Codex. This is apparently documented in PR #1884.

[Josselin Perrus @nonils]

- A Skill would be very, very, helpful for those less technical people I'm a part of