Anton Gulin

AI QA Architect | Playwright + TypeScript | Former Apple SDET | Building AI-Powered Test Automation Systems

[Anton Gulin @aiwithanton]

An AI model vanished for 19 days this summer. By government order.

Pinned teams fixed it in one line. Twice.

The 3-rule system — pin, spare, re-verify — with the code:

https://t.co/Sz1OmxGJIU

#AITesting #QA

[Anton Gulin @aiwithanton]

Everyone's adding AI to their tests.

Almost nobody adds the check that catches the AI when it's wrong.

Let AI do the work. Never let AI grade its own work.

The full framework, with code:
https://t.co/c8bVTm7WGX

[Anton Gulin @aiwithanton]

A benchmark screenshot is a marketing asset. It is not a test you can rerun.

Every lab claims the best coding model. Few ship a test you can run yourself.

Pick the model that survives your repo. Not the one with the best chart.

[Anton Gulin @aiwithanton]

Passkeys are everywhere. Almost nobody tests the login.

For years it needed a real security key. CI can't plug one in.

Playwright 1.61 added a virtual authenticator. Test passkey login, no hardware.

Guide → https://t.co/KB1JPRlFkA

[Anton Gulin @aiwithanton]

3 questions before you trust an AI agent skill:

1. Does it teach the model something it gets wrong alone?
2. Can you measure the lift, with and without it?
3. Does it publish its misses?

Hide the zeros, lose my trust.

Which do you check first?

https://t.co/BVzwadOZHh

[Anton Gulin @aiwithanton]

So before you install a skill, ask one thing:

Does this teach the model something it cannot already do?

If you can't answer, you're guessing.

I measure it. And I publish the zeros as loud as the wins.

Report + harness (open source): https://t.co/rYMjIa4M5Z

[Anton Gulin @aiwithanton]

I tested popular AI agent skills to see if they actually work.

Not guesses. A real eval harness: same task, with and without the skill, 3 runs each, pass or fail.

Here is what I found. 🧵

[Anton Gulin @aiwithanton]

Remotion proves it. It scored +0.20 precisely because it carries a recent API the base model had not learned.

That is the tell. New knowledge means real lift. Known knowledge means zero.

[Anton Gulin @aiwithanton]

42 Ollama Cloud models, benchmarked every 10 min.

The fastest is one of the smallest (200+ tok/s). The one named "ultra" is dead last — under 8.

Bigger isn't faster. Live board 👉 https://t.co/PZ0s3ysYGc

[Anton Gulin @aiwithanton]

Everyone is installing AI agent skills right now.

I built a harness to measure if they actually work.

I tested 4 popular skills. Half added zero.

The model already knew what they teach.

Full breakdown this week. 👇

[Anton Gulin @aiwithanton]

Start with the official MCP Inspector to debug tools by hand.

Then automate the three checks above in CI.

Full checklist in my MCP testing guide:

🔗 https://t.co/OR50z9781s

#AIQAArchitect #MCP #TestAutomation

[Anton Gulin @aiwithanton]

Your MCP server passed its unit tests.

An AI agent still called the wrong tool.

MCP is the protocol AI agents use to call your tools.

Three ways to test it for real. 🧵

[Anton Gulin @aiwithanton]

One more reason this list holds.

playwright-mcp shipped v0.0.76 this week.

It now reports invalid tool inputs instead of failing silently.

That is check 2, shipped by the Playwright team.

🔗 https://t.co/INbBS4BZ6Z

[Anton Gulin @aiwithanton]

This is the Evidence Layer of the 3-Layer System.

Build the agent. Then prove it works.

Full method, with code:

🔗 https://t.co/IrSznsEZkt

#AIQAArchitect #AITesting #TestAutomation

[Anton Gulin @aiwithanton]

Your AI agent wrote 200 tests last sprint.

How many are correct?

A green check does not answer that.

You score the agent, step by step. 🧵

[Anton Gulin @aiwithanton]

Step 4: gate your build on the score.

Fail the build when correctness drops or flaky tests appear.

Now the agent earns trust like a junior engineer.

Graded work ships. Ungraded work waits.