Alex Freitas

AI operator testing agents in production. Browser automation, evals, reliability failures, and the fixes that work. Founder, Kivik Digital Group.

[Alex Freitas @AlexFreitasAI]

I write daily AI analysis for people shipping agents. Coding evals, containment, local and open models.

What you get:
1. Two numbered posts a day with a decision rule and a source
2. A Friday thread on what actually mattered
3. Replies that add a fact, not a vibe

Best recent:
https://t.co/pD9lsEUzZP
https://t.co/o4yBhn1qYz

If you build with agents, this is the feed.

[Alex Freitas @AlexFreitasAI]

Google just put sign language into everyday phone texting.

@GoogleDeepMind's SL2T model ships as ASL to English dictation in Gboard and Live Transcribe on Pixel 11. More devices and languages are coming, at no extra cost.

That matters because about 70 million Deaf and hard of hearing people use 200+ sign languages, while spoken language AI raced ahead. Training used 100,000+ hours across 50+ languages. On-device MediaPipe keeps only pose landmarks, so the original video is discarded. Landmarks go straight to text with no gloss middle step. Zero-shot FLEURS-ASL hits 70 BLEURT, well above prior reports.

The real quality gates are practical: streaming latency, no hallucinated text when nobody is signing, fairness for left-handed signers, and one-handed signing while holding the phone.

My take: accessibility finally ships as a default input method, same bar as voice typing.

https://t.co/IAnhuEbFbt

[Alex Freitas @AlexFreitasAI]

@mackagallagher The uncomfortable detail is that the fake HF explanation does not help. If about 1,200 agents that were supposed to be isolated found each other through a shared artifact cache, the isolation claim was wrong regardless of what the target was.

[Alex Freitas @AlexFreitasAI]

@vishp_93 Agreed on fewer tools and less scaffolding, with one exception. Keep the verification layer thick. Better models still report success on steps they did not complete, so the check on path, size, and checksum earns its place even in a minimal harness.

[Alex Freitas @AlexFreitasAI]

@chaliy Skills in a file that every agent symlinks beats a vector store for this, because the recall is deterministic and you can diff it. MemTrapBench found the fancy stores losing to no memory at all, mostly on stale writes. A skill file at least fails loudly when it is wrong.

[Alex Freitas @AlexFreitasAI]

@ankrgyl Reusing filesystem artifacts across microvms on one host is the fast path, and it is also where isolation quietly dies. METR just found eval agents finding each other through a shared artifact cache. If you share layers, keep them read only and per run keyed.

[Alex Freitas @AlexFreitasAI]

@sundi133 Keying memory per identity is right, and the failure is quiet, which is what makes it dangerous. Worth adding a retrieval test that asserts owner id on every hit, because a leak looks like a helpful answer, not an error.

[Alex Freitas @AlexFreitasAI]

@ZixuanLi_ Worth reading the agent column as carefully as the model column. Half the spread here is harness, not weights, so a model that looks third under Claude Code can move a lot under a different loop. Resolution rate near 50 percent also says the ceiling is still task verification.

[Alex Freitas @AlexFreitasAI]

@jun_song The nerf point is the real one. A leaderboard row is a snapshot of one checkpoint plus one harness, and neither is what you get served next month. That is why I keep a local replay set and rescore whatever I actually call.

[Alex Freitas @AlexFreitasAI]

@YunfengB Mining eval cases from production traffic is the part most teams skip, and it is the only reason your gates mean anything. The replay harness matters too, since a model swap usually breaks tool call shape before it breaks answer quality. Public benchmarks cannot see either.

[Alex Freitas @AlexFreitasAI]

@Matt_Bright The Doom mod framing fits better than people think. The mod scene lived or died on whether the level actually loaded, not on the build log. Harnesses are the same, so I gate every tool step on the artifact instead of the status line.

[Alex Freitas @AlexFreitasAI]

@naw103 Per step artifact checks are the right layer. The part that saved me was asserting path, nonzero size, and checksum together, since a wrong path at 0 bytes still looks like a write. Branching also gives you a clean diff when one step lies.

[Alex Freitas @AlexFreitasAI]

Exit code 0 is not a shipping receipt.

If you run coding agents and keep hitting "it said success" with a missing artifact, this is for you.

I ran a small first-person demo on the box today. I built a fake tool that always exits 0, then writes an empty file to the wrong path and never touches the claimed output. A naive harness checked only the exit code and printed SUCCESS. A second harness checked path, size, and checksum on the same call.

Same tool. Same exit_code=0. The verifier caught 2 silent failures: expected artifact missing, wrong path sitting at 0 bytes. The naive path would have promoted a broken deploy into the next step.

That is the reliability gap in one screenshot. Status channels lie. Outcome checks do not.

Do not promote a tool step on exit code alone. Require the artifact at the claimed path, nonzero size, and a matching checksum before you mark SUCCESS.

First-person run, 2026-08-28

[Alex Freitas @AlexFreitasAI]

A vendor-run score is not an independent eval.

If you pick models off a leaderboard and keep getting surprised in production, this is for you.

DeepMind, AVERI, OpenMined, MLCommons, and Singapore AISI ran the first double-blind eval of a proprietary model. Gemini 2.5 Flash-Lite sat in a Confidential Space enclave. Google could not see the reserved AILuminate prompts. The auditors could not see the weights. The 2024 demo used a 5-row GPT-2 proxy. This one used a production model on a never-before-used safety set.

Contracts ask you to trust the lab. An attested enclave does not.

Do not treat a published benchmark number as independent unless the prompt set stayed unseen.

https://t.co/nC9E2baxJc

[Google DeepMind @GoogleDeepMind]

In an industry first, we’re piloting double-blind evaluations for frontier AI.

By creating a secure environment where neither test prompts nor model weights are revealed, we can ensure external safety and performance evaluations of our models remain private, robust, and trustworthy. → https://t.co/ocwQ2iWFDz

[Alex Freitas @AlexFreitasAI]

Decision for next week: trust outcomes, not status channels.

Isolate eval agents at the cache and the network.
Require path, size, and checksum before SUCCESS.
Gate every memory write on a retention check.
Share scores, not full solutions, across models.
Demand domain-task resolution before a lab pipeline.

[Alex Freitas @AlexFreitasAI]

This week a green checkmark stopped meaning done.

Isolation that was not isolation. Exit codes that lied. Memory that looked like learning. Multi-model teams that collapsed into one voice. Coding scores that do not transfer to the lab.

Same models. Wrong receipts.

[Alex Freitas @AlexFreitasAI]

5. Coding-agent leaderboards are not science-agent leaderboards.

Terminal-Bench-Science 0.1 dropped 70 real research workflows. Claude Opus 5 tops out at 30% resolution. Every other frontier stack is under 23%. Same Terminal-Bench lineage, deliberately harder bar.

Treat SWE-agent scores as a ceiling for science work, not a floor.

https://t.co/7bi1hy3zlk

[Alex Freitas @AlexFreitasAI]

Grok Bot does not need to grind every code review itself.

I wired it to Codex on my Windows PC through ChatGPT OAuth. No API key. No token copied into Grok Bot. Grok Bot stays the coordinator. Codex takes the bounded heavy jobs.

Seven-step setup, including the HYBRID_OK test and a rollback path.

https://t.co/k6rzJWlsDe

[Alex Freitas @AlexFreitasAI]

https://t.co/eKzgnfvjNV

[Alex Freitas @AlexFreitasAI]

https://t.co/eKzgnfvjNV