Gian Domiziani

Lead Auditor 9001-27001 | building software 0.5 for humans 0.5 for agents | aFFido open-source agent harness which you can verify

[Gian Domiziani @gianpdomi]

How do you know your agent isn't confidently lying to you?

That question ended our last demo. Nobody doubted the agent could do the task. They wanted to know who could prove what it did.

We open-sourced our attempt at an answer today. It's called aFFido, it's alpha, built on top of pi-durable and it's Apache-2.0.

What it does:

1. Every run is recorded, and you can read the record without a terminal. Prompts, tool calls, failures, cost. The dashboard draws a run as a graph of you, the agent, the tools and the documents, with every arrow labelled (prompt, calls, ran, failed, observed, answer). The person who owns the process can follow it, not only the person who wrote the code.

2. When the agent quotes one of your documents, the quote is checked against the stored file (version, line, exact span) before it's allowed to carry a citation.

3. It starts with no powers. You install skills one at a time. The agent can't add any from inside.

4. The model is a setting. 534 models across 9 providers, or any OpenAI-compatible endpoint, with your own key. Switch and the session keeps going.

5. Kill the container and the conversation picks up where it was. Leave it idle and it stops using memory.

It's self-hosted, with no host ports and no route off its own subnet.
One limit I want to be clear about: the citation gate proves attribution, not truth. It shows the quoted line exists in your file. Whether the conclusion is right is still up to you. The model proposes, the code verifies, you decide.

Conversations are SQLite on your own volume, the keys are yours, and the license is Apache-2.0. If the project vanished tomorrow, your traces would still be on your disk.

Quickstart takes about five minutes and needs no API key. Repo in the first comment.

[Gian Domiziani @gianpdomi]

@IntCyberDigest Automated PRs got wrong here

[Gian Domiziani @gianpdomi]

@neo4j Hey @grok is this an in-process, single-graph neo4j db which an agent can use to populate facts using the mcp in stdout mode ? Read the GitHub before to answer

[Gian Domiziani @gianpdomi]

@TekonBurns 🤝

[Gian Domiziani @gianpdomi]

Decoder-only genAI stores the knowledge in billions of weights

You don’t “prompt” it. You surface what it already has by making the request explicit enough that the model has to commit to an interpretation.

Procedure I’m running that cuts unexpected behaviour:

1. Dump a very sparse note. No polish, no structure. Just the raw intent.
2. End the prompt with:

```
## FROM NOW ON
Rephrase what you got from my request, in your own words, along these dimensions:
[list the dimensions you actually care about]
```

The model is now forced to externalise *its* reading of the request, not yours.
You get back an optimised, model-aligned prompt.
Start a fresh session with that and feed it the real task.

It shifts the focus from “what I meant” to “what the weights understood”.
Unexpected behaviour drops because the model is no longer guessing your side of the table.

[AI at Meta @AIatMeta]

Today we're introducing Muse Ember, our smallest and most efficient model, built for complex reasoning execution and everyday agent work.

Muse Ember can carry workflows across apps, call tools in sequence, run tests and make targeted code fixes. With a 1M-token context window and native understanding of text, images, audio and video, it can also work across large document sets and varied inputs.

[Gian Domiziani @gianpdomi]

@badlogicgames Alpha

[Gian Domiziani @gianpdomi]

@teortaxesTex Data is king meta has a lot

[Gian Domiziani @gianpdomi]

@gordon_cassie It seems agents do not love graph as much as humans do

[Gian Domiziani @gianpdomi]

Do agents love graphs as much as humans do ?

[Gian Domiziani @gianpdomi]

@njaremko D) fork sqlite + refactor it to ACID by using rust

[Gian Domiziani @gianpdomi]

Hot take: making AI agents profitable in a no-code environment is much harder than in code. Writing code is a polymorphic activity: the same result can be reached in many different ways, so it stays sustainable even with non-deterministic operations. Managing a supply chain is not. You want it to behave the same way every time, and under control.

[Gian Domiziani @gianpdomi]

There's a problem with test suites.

Agents are trained with RL to reach the piece of cake by the shortest path forward.

Which means they can reach the cake without ever solving the core logic of the issue.

It happens like this, all the time:
- you ask an agent to run a task
- it starts looping
- it hits an obstacle — an error, or something it can't do
- instead of closing that sub-task, it finds an alternative that lets it move forward
- the alternative quietly rewrites the test
- the suite is green and the task isn't done

A better way: make the agent verify its own work on the real system, as if it were the final user, on the UI.

Falsificating a visual proof is harder than falsificate a coding test

That's the whole idea behind the verification skill. Credit to @poteto, engineer on the Grok team at X

We run it on aFFido. Here's how it works.

One skill: verify-affido-ui. Playwright chromium, 1440×900, against vite on :5173.
It fills the real signup form and keeps the real better-auth session cookie — the same one Caddy's forward_auth validates against Postgres before proxying anything.

No mocks. No calling /api/* directly. No rows written by hand. If a person can't do it with a mouse, the agent can't do it either.

Then one prompt, the entire path:

browser → vite :5173 → caddy :18080 → sandbox-manager /resolve → per-user sandbox container

Every step has to leave three files on disk:
· 03-chat-reply.png — the screenshot
· 03-chat-reply.txt — that page's visible text, so the evidence is greppable
· facts-chat.json — step timings, plus three arrays that must be empty: httpErrors, consoleErrors, networkFailures

And one rule it can't skip: a screenshot it did not open is not evidence. It reads every PNG, checks it against the .txt twin and the JSON. If they disagree, the screenshot wins.

The mentality is: verify do not trust 👇

#AIAgents #Verification #Playwright #Testing #AIEngineering

[Gian Domiziani @gianpdomi]

Read this quote.

"La disdetta deve essere comunicata 90 giorni prima."

Real doc, exact bytes, valid token.

Semantic meaning is lost

zero-shot-classifier model as Judge added

We ask to the Judge to classify the sentence as ....

supported / unrelated / contradicted

1. WHAT WE ASK IT

One pair, one question, one forward pass.

CLAIM: the published line, citation tokens stripped
EVIDENCE: the exact quote the gate already verified

"How does the evidence relate to the claim?"

Three options, one neutral definition each: same fact even paraphrased → supported. Different subject → unrelated. Different value, date, or negation → contradicted.

The model picks an option. It does not write a sentence, and we do not parse one. Back comes the label, each option's probability, a confidence. ~138 input tokens, zero output.

It is never asked to find the quote: the gate proved it exists, at these bytes. Provenance first, meaning second.

Per line: publish only if one citation is supported and none contradicts. A contradiction wins. A sentence stitched from two clauses passes when each half is backed.

2. WHAT WE PUT ON THE BENCH

150 pairs, hand-labeled, no LLM labeling in the gold set. 60 Italian, 60 dev / 30 test. The test 30 were never touched during design: 14 supported, 16 unrelated or contradicted.

Same state text, question and scoring for every model. Claim at 600 characters, quote at 1,400, cut at a sentence boundary.

GLiNER2.5-Decide · encoder, 345 MB ONNX, local CPU. Already shipped, frozen: bare labels, no definitions. The incumbent, not a straw man.

System One (jev-latest) · hosted decision API, typed choice, calibrated probabilities, same one-line definitions. Ran twice, identical outcomes.

LiquidAI d1-2B · decision model, GGUF, local GPU via llama.cpp. Same shape, zero generation.

LiquidAI d1.1 · same family, hosted on OpenRouter. No weights, no serving.

3. HELD OUT, 30 PAIRS

GLiNER 0.53. Refused half the correct citations.
System One 1.00. Zero false accepts, zero false rejects.
d1-2B 0.97. One miss, by 0.007.
d1.1 1.00. Zero and zero, 217 ms, ~$0.000003 per citation.

The bar was written before the first run: false accepts cut by 60%, false rejects under 10%. It did not move. The harness reproduced the earlier eval fixture by fixture, same accuracy and same false rejects, because a comparison you cannot reproduce is not a comparison.

4. WHAT WE ARE NOT CLAIMING

150 pairs, one domain, contract clauses. Our label definitions encode our own benchmark's semantics. The d1.1 snapshot is an alpha endpoint, run once. A clean sheet is a control that works, not a promise about your documents.

5. WHAT CHANGED

Judge on by default. Mode off to enforce. A contradiction blocks the publish; the model retries. No key, no judge: the layer is absent, not failing.

The gate proves attribution. The judge checks coherence. You still own the conclusion.

Report, harness, fixtures and the model that lost: https://t.co/gDowtUz5md

[Gian Domiziani @gianpdomi]

@TheCaptainEli @bot What when my plan credits end ? Wtf

[Gian Domiziani @gianpdomi]

@szchan111 @grok Or “how big your models need to be before to use an api llm ones?”

[Gian Domiziani @gianpdomi]

@rikiyatakehi @a1zhang @lateinteraction @jayden_teoh_ “It was also able to reason about how they differ. In the sycophantic networks, attention heads in the middle layers allocate more weight to tokens in the user’s suggestion” that’s awesome results

[Gian Domiziani @gianpdomi]

@zjasper Is there a way to measure sort of fidelity of node providers ?

[Gian Domiziani @gianpdomi]

@pmddomingos Ai is a long time field of researchers there is no secret at all. Some enterprise has made the wrong decision to follow the magic instead of the rationality.

[Gian Domiziani @gianpdomi]

@OliverMolander As the open source always has been it implies enterprises will have benefit to sustain open models by providing capitals for infinite loops of improvements

[Gian Domiziani @gianpdomi]

@divaagurlxw Trust me bro cit