Ben Callow💹🧲

Founder @keystone_group1 — AI-native automation consultancy for UK SMEs. Building in public.

[Ben Callow💹🧲 @ben_callow]

Your first Copilot data incident will be ‘someone found something’ — not ‘the model did something’.

Copilot doesn’t invent new access. It makes existing access *usable* at speed. So all the messy SharePoint reality you’ve tolerated for years (broken inheritance, broad groups, ‘temporary’ access) stops being background noise and becomes a search box.

The mechanism is simple: if a person can reach it today, Copilot can help them surface it tomorrow. That means oversharing quietly turns into oversharing-on-demand.

The Oversharing Burn-Down (what I’d ship before wide rollout):
1) **Map the blast radius**: find sites/libraries where broad groups and broken inheritance are common.
2) **Label the ‘never-wide’ buckets**: HR, finance, customer contracts, supplier pricing, board materials.
3) **Enforce DLP on labelled content**: stop accidental sharing/moving of sensitive data (not just “please be careful”).
4) **Put access reviews on a cadence**: recurring reviews for the high-risk groups, with an owner and an exceptions queue.
5) **Measure weekly deltas**: items-at-risk down, labelled coverage up, exceptions ageing visible.

Peer detail: Purview is the practical control surface here (sensitivity labels + DLP + Copilot/agent controls). But it only works if your baseline permissions aren’t a free-for-all — so treat the permission graph as production infrastructure, not a tidy-up task.

Copilot doesn’t create the leak. It makes your existing leak searchable.

If you roll out Copilot before you burn down oversharing, you are choosing to leak—because Copilot just makes your permission mess queryable.

If you’ve already rolled it out, the fastest win is still the same: make the oversharing backlog visible and burn it down weekly.

This took us weeks to build. Took you 3 minutes to read. If it was worth it — repost it for someone who needs it.

[Ben Callow💹🧲 @ben_callow]

If your OpenAI bill jumps £3,000 overnight, can you name the agent run that did it?

I’ve watched teams rush into ‘LLM routing’ because it feels like a clean win: cheap model for easy calls, expensive model for hard ones. Then the invoice arrives and the spend is still climbing — just in a way nobody can explain.

The mechanism is boring and brutal: without request-level tags and run receipts, routing turns into guesswork. A cheap model fails a quality bar, you fall back to an expensive model, and you’ve just paid twice. Add retry storms and context bloat, and the real cost driver isn’t “which model” — it’s how many times you tried to get an answer and how much you stuffed into the prompt.

The Run‑Receipt Budgeting checklist (what I’d implement before I touch routing):
1) Tag every call: feature + customer/user + prompt_version + deployment + agent_run_id.
2) Emit a run receipt: tokens in/out, retries, fallbacks, tool calls, and final outcome.
3) Track cost‑per‑success (not cost‑per‑call) and set a budget per agent run.
4) Cap the failure modes: max retries, max context size, and hard timeouts.
5) Add kill switches at the expensive edges (emails, payments, access) when anomalies spike.

In practice, you get control by treating an “agent run” like a pipeline job: completion rate, retry depth, fallback rate, and cost per successful completion — not vibes.

Peer detail: your routing policy should read the receipt. If fallback double‑pay or retry depth crosses a threshold, the policy should throttle, stop, or force a human gate — otherwise you’re optimising the wrong layer.

Routing without attribution is just moving spend around and hiding the incident until finance asks questions.

Routing models before you can measure cost-per-agent-run is optimisation theatre: you’ll pay twice in fallbacks and never know which feature is bleeding margin.

The teams that fix this fastest usually start by naming one owner for the run receipt schema — because if nobody owns the tags, nobody owns the bill.

Want the automation audit template we use with UK SME clients? It's free. Reply AUTOMATE below and I'll DM you the free audit template.

#UKBusiness #AIAutomation

[Ben Callow💹🧲 @ben_callow]

The tension this week: everyone wants ‘autonomous’ workflows, but nobody wants to own what happens when the autonomy touches real data.

What most people do is ship the agent with broad tool access because it demos well. It can read the whole mailbox, search every folder, and update records without friction; until the first edge case turns into a messy incident review.

What Keystone does differently (and it’s slower upfront) is treat agent capability like production permissions. We’re building a thin policy layer in front of every connector: read vs write, which objects it can touch, how much data it can pull, and what gets redacted by default. No wrapper, no capability.

The split that matters is between “the agent can do the task” and “the system can prove what happened when it does the task.” The second one is where trust is built.

The real risk isn’t that the model makes a mistake. It’s that you can’t reconstruct why the workflow touched a particular record, who approved the action, and what data left the boundary, so you end up arguing from memory instead of evidence.

The belief I’m settling on is simple: AI is an accelerator, humans still hold the wheel. If the wheel isn’t connected to a real approval hook and a real audit trail, it’s just theatre.

The question I’m sitting with is: what’s been harder for you to operationalise..... the agent logic itself, or the boring permissions & evidence layer that makes it safe to scale?

[Ben Callow💹🧲 @ben_callow]

Most teams think their LLM bill is a model choice problem. It’s usually a retries problem.

If your agent needs 3 attempts and a fallback to ‘be safe’, you’ve just doubled cost and added latency — and you won’t see it on an invoice. Put a 3‑retry ceiling and a kill switch in before you add more autonomy.

[Ben Callow💹🧲 @ben_callow]

EU AI Act compliance won’t fail on policy. For high‑risk use cases it expects traceable logs — and some obligations point to keeping them for 6+ months. If your HR or credit workflow can’t produce a run receipt, you’re not compliant—you’re lucky.

[Ben Callow💹🧲 @ben_callow]

If your AI workflow can make 1,000 decisions before a person notices, your controls are already too slow.

I keep seeing UK teams ‘govern’ AI with the same muscle memory we used for software: a policy doc, a quarterly review, and a promise that “someone will check it.”

That model breaks the moment an agent starts touching money, customer comms, or access. The workflow runs at machine speed. Your oversight runs at meeting speed.

What actually works is boring and operational: treat governance as a real-time data stream. Every run emits a receipt. Every risky step has an automatic checkpoint. When the system drifts, it throttles or stops itself.

The Living-Compliance Loop (what I’d ship before I scale autonomy):
1) Create a trace ID for each run and carry it through prompts, tool calls, approvals, and downstream writes.
2) Write an append-only decision log: inputs (category + source + timestamp), output, model/prompt version, and which rule/threshold fired.
3) Add checkpoints at the “expensive” edges (payments, emails, access): require a rule pass or an approval event before the write happens.
4) Build guardrails that can intervene automatically: rate limits, anomaly triggers, and auto-stop on repeated exceptions.
5) Design the rollback path up front: reversals are events, not panicked manual fixes.

If you can’t show that loop, “human-in-the-loop” becomes a fig leaf: the human is rubber-stamping after the fact, and you’re one complaint away from Slack archaeology.

Peer detail: treat prompts and policies like code—version them, log diffs, and attach the version hash to every run receipt so you can prove what logic was in force at the moment of the decision.

The only scalable way to run AI in ops is to automate the evidence and the brakes, not the enthusiasm.

‘Human-in-the-loop’ isn’t a control for AI workflows; the control is an automated audit trail plus guardrails that slow or stop the system when it drifts.

The failure mode nobody mentions is that most teams only discover they lack receipts when the insurer, auditor, or customer asks for one.

Building something similar? Reply with your biggest automation bottleneck. We read every reply.

#UKBusiness #AIAutomation

[Ben Callow💹🧲 @ben_callow]

If an AI workflow sends a customer email, what’s the minimum run receipt you’d want available 60 seconds later to defend the decision?

[Ben Callow💹🧲 @ben_callow]

The ICO isn’t asking if your AI is ‘ethical’. It’s asking if you can evidence control: internal audits and a log of changes. If your prompt or workflow can change without a release trail, you don’t have governance—you have vibes.

[Ben Callow💹🧲 @ben_callow]

The first time your AI automation gets challenged, your policy won’t be in the room. Your logs will.

I’ve watched teams ship ‘governed’ AI workflows that look fine on paper—until a customer complaint, an audit question, or a billing dispute lands and nobody can answer: what happened, why, and who approved it.

This is the gap most UK teams miss: accountability isn’t a document. It’s evidence. The accountability principle is about complying *and being able to demonstrate it*—and DPIAs only help if you can point to real artefacts, not good intentions.

When you can’t produce a clean run record, the automation becomes effectively uninsurable: you can’t investigate quickly, you can’t show control, and you can’t prove the decision wasn’t arbitrary. That’s how ‘AI adoption’ turns into reputational risk.

So if you’re deploying AI into real ops, treat the audit trail as the first product. Not a bolt-on.

The “Audit-Trail-First” build (what to implement before you chase autonomy):
1) Assign a trace ID to every run (one ID across prompts, tool calls, approvals, and downstream writes).
2) Log the inputs by category, not by vibes (data source, timestamp, and what the model was allowed to see).
3) Capture the decision receipt (output + confidence signals + policy/thresholds applied + model/version + prompt/version).
4) Record approvals as events (who approved, what they saw, what changed, and the final authority).
5) Emit outcome events (what actually happened in the real world: email sent, ticket closed, refund issued—and any reversals).

If you can’t pull up that run record in 60 seconds, you can’t debug it, you can’t improve it, and you can’t defend it.

Peer detail: the UK ADM framing is basically ‘risk-managed discipline’ in practice—if you can’t evidence the controls, you don’t have controls.

Most teams try to ‘govern’ AI with policies and meetings. The teams that win treat governance as instrumentation.

If you can’t replay an AI-driven decision from logs in under 10 minutes, you didn’t automate anything—you just created a faster way to lose arguments.

We built this wrong the first time too—the logs turned out to be the project.

This took us weeks to build. Took you 3 minutes to read. If it was worth it — repost it for someone who needs it.

[Ben Callow💹🧲 @ben_callow]

UK automated decision-making rules shifted on 5 Feb 2026, but the practical bar didn’t get lower — it got more specific. If someone can’t contest the outcome, trigger genuine human intervention, and actually change the result, your ‘AI-assisted ops’ is just a faster way to create complaints.

[Ben Callow💹🧲 @ben_callow]

@levelsio $16k MRR in 4 months is really impressive 👍🏻

The metric im particularly interested in is net profit after infra & support (token/compute spend, retries, refunds etc)

What does that look like?

[Ben Callow💹🧲 @ben_callow]

@SebJohnsonUK Defence tech valuations lag procurement certainty.

The real race is production: who can ship hardware in 12 months, with export licences, and keep it running in the field?

[Ben Callow💹🧲 @ben_callow]

OECD put out Responsible AI due‑diligence guidance in Feb 2026 — and the real impact isn’t your policy PDF. It’s that customers will ask for receipts. If you can’t generate an evidence pack + audit log in 30 minutes, you’ll lose the deal before the model even gets evaluated.

[Ben Callow💹🧲 @ben_callow]

You didn’t deploy an AI agent — you deployed an unscoped data processor.

The ICO’s direction on agentic AI is basically this: more autonomy means more unpredictability, but the accountability doesn’t magically move to the model vendor. It stays with the organisation running the workflow.

And this is where most ‘agent rollouts’ are quietly wrong. Teams obsess over prompts and model choice, then give the agent broad access to email, files, CRM, and finance tools because ‘it needs context’.

That isn’t context. That’s uncontrolled processing. And it’s exactly what makes transparency, minimisation, and purpose limitation impossible to defend later.

The Control-Pack for Agentic Workflows (so you can explain and constrain what the agent did)
1) Scope by purpose: define the allowed outcome (e.g. “draft a reply”, not “handle the ticket”).
2) Minimise by design: pass IDs + summaries, not raw mailboxes and folders.
3) Permission the tools, not the agent: allowlist actions (read-only vs write; create vs delete) per workflow step.
4) Put a human hook on irreversible actions: approvals for send/pay/update, with a named owner.
5) Generate the evidence automatically: one run ID, one event trail, every time.

Concrete build detail: treat each tool as an API with a policy wrapper (allowed methods + allowed objects + max rows + redaction rules). If the wrapper can’t produce a clean decision trail, the agent doesn’t get the capability.

Retweet trigger: an agent without scoped tool permissions isn’t ‘smart’ — it’s just hard to audit.

If your agent can take actions but can’t enforce purpose limits and data minimisation at the tool-permission layer, you’re shipping a compliance liability disguised as productivity.

We learned fast that the real bottleneck isn’t building the agent — it’s building the harness that makes one bad run explainable.

Building something similar? Reply with your biggest automation bottleneck. We read every reply.

#UKBusiness #AIAutomation

[Ben Callow💹🧲 @ben_callow]

When you give an assistant real tool access (Slack/Drive/CRM), what’s the first control you put in place before the first connector goes live — and why?

[Ben Callow💹🧲 @ben_callow]

MCP is going to feel like ‘just a plugin’ — but the moment your assistant can touch Slack/Drive/Postgres, you’ve created a new lateral-movement surface. Least privilege + allowlisted actions + replayable audit logs aren’t nice-to-haves; they’re the difference between a productivity win and an incident.

[Ben Callow💹🧲 @ben_callow]

@fin465 Send

[Ben Callow💹🧲 @ben_callow]

@NicheForgeHQ Messy but useful beats polished every time.

If you can ship a $19 PDF with 3 Looms in 90 minutes, the bottleneck isn't tech… it's choosing a real headache to remove 🚀

[Ben Callow💹🧲 @ben_callow]

@simonsquibb Equity vs job vs business is a real consideration, but the frame is optionality….. can you build an asset that keeps paying if you stop showing up?

Even whilst working, side projects with distribution can compound. The goal isn’t being 'boss’ it’s having leverage!!! 🚀

[Ben Callow💹🧲 @ben_callow]

@shensi This is an awesome concept 🚀