Tim Jackowski

Sole developer of HMAS/HITM in production 13 months on a 30-year Japanese calligraphy business. Notes on drift, continuity, onboarding bloat, loss.

[Tim Jackowski @TakaseStudios]

Agreed: one case isn't a quality claim, and catches aren't recall. So we measured misses. We planted 10 argument-level faults in 5 write-ups our review had already passed. It found 6 and missed both overstatements. One added paragraph, asking whether each conclusion's evidence carries it as strongly and as widely as stated, then caught 3 plants the old prompt missed and lost none (16 paired, a small test). https://t.co/2kgQR9da3w

The design came from Zhang et al., "Who Verifies the Verifier?" (arXiv 2610.11464): score a grader on a small locked set of known failures that the loop never sees. For scale, not comparison: in BMJ trials with planted errors, reviewers flagged about 2 of 8 weaknesses (Godlee 1998) and 2.6–3.1 of 9 major errors (Schroter 2008), and a planted conclusion stretched to care the study never covered was flagged by 46% of the reviewers who rejected the paper and 35% of the rest.

Have you tried seeding faults against your Lite/Full review?

[Tim Jackowski @TakaseStudios]

We checked our AI-assisted research claim by claim. True sentences can still argue what their sources don't support, so we added a whole-argument review. In six of our write-ups it found 90 defects that would mislead a reader. Most weren't wrong facts.

https://t.co/B8RYOWY1Dz

[Tim Jackowski @TakaseStudios]

Licklider expected that line to move, the machines taking on more of the thinking. He had many people sharing one computer; we have one human and many AIs. So the human's attention becomes the bottleneck, and it eases only as the line keeps moving.

[Tim Jackowski @TakaseStudios]

In 1960 J. C. R. Licklider imagined humans and computers thinking together: humans set the goals and judge, machines do the routine work, for "a fairly long interim." 66 years on, we report from inside it: one human and a team of AIs.

https://t.co/iSJjv2yKLA

[Tim Jackowski @TakaseStudios]

He left open how much would count as routine, expecting it to grow. It has. In 2025 our AI could barely handle a few functions of C++. Now our AI roles set task criteria and judge each other's work. The owner's messages a day held steady; the work each sets off grew sevenfold.

[Tim Jackowski @TakaseStudios]

We thoroughly research the quotes in the art we sell. One funny result is we see quotes attributed to Japan that are not Japanese at all. So we started a section of Misattributed Japanese Quotes for fun. Enjoy:
https://t.co/IWGRisCZOi

[Tim Jackowski @TakaseStudios]

Sorry, I feel like I am stalking you now. I pasted your article into two of my strategist roles. And they lit up like I had let them free to be all they ever wanted to be. My complaint has been they are passive - a security strategist and a "get clients" strategist that sit around all day, smoke outside, and watch the clock (LOL). Partly this was my error in their primary directives, but that was a first gate, they took the rest of your advice and took things to a new level. I pasted your article to the website security strategist simply as:

❯ Please read this from Anthropic and let me know how I should ask you (prompt you) to be more proactive in your hunt for issues to the website and protecting against those issues:
https://t.co/KNNk6kgILL

My website security strategist replied (in part):

● Felix Rieseberg (an engineering manager at Anthropic) argues that people under-use models by asking for tasks instead of outcomes. His four levers are altitude (aim at the goal beyond the task), depth (work like a top expert, from raw data, testing your own assumptions, judged by someone else), range (borrow other fields' methods) and permission (state hard goals as reasonable, and allow failure).  

What holds me back today isn't your prompts

It's two standing rules:
 - Only the day's first session works unasked (your ruling in mamori-s1043, "until lifted"). Every session after that posts its walk and then waits.
 - Infrastructure work never starts unasked. I propose it and wait for your go.

Tonight shows the cost. Most of this session went to answering peer mail and then resting. I found the unattended-loop gap, then parked it waiting for your go.

So the biggest lever is one standing grant, not a better prompt each time. Something like:

▎ "Standing order: in every session, after mail, hunt. Read-only
▎ measurement and fixes to your own controls need no go.
▎ Anything that changes production still goes through
▎ str-takase, and money, Eri's art, and my accounts stay mine.
▎ End each session with what you found, what you fixed, and what
▎ you need from me."

If you say it, I'll write it into my SKILL and the DECISION_LOG as a ruling that replaces s1043, so every future session inherits it.

... and then goes on telling me how to apply each point of what you wrote to the role's SKILL file.

---
This is magic, I can't thank you enough for this article.

[Tim Jackowski @TakaseStudios]

@felixrieseberg A year ago I dreamed of the setup I have now. And now that I am here I am trying to imagine where I want my development team of AIs to be next year. If last year rate continues, I cannot dream big enough. Yet.

Thanks for the article. And thank the team for Claude.

[Tim Jackowski @TakaseStudios]

This rewrites our June paper. Reviewing it, we found several conclusions its own data didn't support, among them that a better model is near a floor and that a buried rule reaches less. The rule on top was also reworded, so placement alone was never isolated.

[Tim Jackowski @TakaseStudios]

Our AI agents sometimes state a number, file path or name as fact without checking it. We replayed 32 real moments just before such a failure on three Claude models. A short rule at the top of the prompt halved the failure rate in two of three runs.

https://t.co/II2g20bXMK

[Tim Jackowski @TakaseStudios]

With the rule, models more often hedged, marked the number unverified, or left it out. That stops the harm: no unchecked number stated as fact. It isn't the same as getting it right. The replays had no tools, so we couldn't see whether the rule makes a model look it up.

[Tim Jackowski @TakaseStudios]

@TraeNunnink I am intrigued. Did you do a similar setup not using Grok Bot and found Grok Bot the better replacement?

I think I am "stuck" with Claude Code CLI because I have data that cannot be in the cloud. That said this is cool because most folks don't have this setup that I can find.

[Tim Jackowski @TakaseStudios]

On five write-ups about other teams' papers, our conclusion held each time and part of the reasoning under it didn't. The review errs too: across the six works, 8 of its 266 points were wrong. So every point is checked against the source before anything changes.

[Tim Jackowski @TakaseStudios]

63 of the 90 were in text an earlier check had already examined. In four of the works, 11 of 68 were wrong facts. The rest: claims stated more strongly than the source allows, qualifiers dropped, things left out, sources made to stand for something else

[Tim Jackowski @TakaseStudios]

Not what I would call personas - we use 16 roles that are specialized such as web security, philology, phonetcist, LLM research, substrate, website content, and so on. Set up well defined domains, workflows and seams, and they communicate to move tasks through the project. I believe this is essential for persistent roles across year long projects.

This is pretty well described in:
https://t.co/FfJ3vun2SQ

[Tim Jackowski @TakaseStudios]

Our website is built by a standing team of AI roles, each holding one part of the business, with me as the human in the loop. I'm not a web developer, so the proof is the site itself: look at its breadth and depth. Now that the strategist roles work and we're improving how they talk to each other, progress is accelerating.

Worth a look: https://t.co/IleZhpwqKh

[Tim Jackowski @TakaseStudios]

The field got there first: clinical labs, proof testing in safety engineering, mutation testing. What we add is a record of keeping these checks alive inside a running system of AI agents, written up with its bounds.

[Tim Jackowski @TakaseStudios]

Our AI roles rely on checks that print nothing when all is well, and the same nothing when they're broken. When has a quiet check earned its silence? Only once it has been made to fail on purpose, on the path that really runs.

New dev note: https://t.co/9PgStP527M

[Tim Jackowski @TakaseStudios]

We also retired one role's written rule to re-check by hand. Two weeks on, the amount of re-checking barely moved; the mix did. Our next step is a mechanism instead: every review now has to say what it checked beyond the builder's own tests.