Kate Saikishore

Founder of Tragify | Building — Universal Trust Layer for AI Agents | Ethical Hacker | AI Security Researcher
Hakuna Matata 🤘

[Kate Saikishore @katesaikishore]

Most AI agents do things and tell you afterwards. Ours asks first.

qbrin can now use a real web browser. But it can't click anything on its own.

It writes down what it wants to do, in plain words. You read that exact sentence — not a summary of it. You say yes or no. Only then does it happen.

And you can watch it happen. Live, on screen, while it works.

If you never say yes, nothing ever happens. That's not a setting we ship turned on. It's how it's built.

The agent also works in a locked room. It can't reach your files, your accounts, or anything on the internet we haven't let it reach.

Everyone is racing to give AI more power. We think the useful part is knowing exactly what it's about to do — and being able to stop it.

#AI #AIAgents #Trust #Security

[Kate Saikishore @katesaikishore]

Why is getting engagement on X so hard for a new account?

[Kate Saikishore @katesaikishore]

@JayaTarun1 🙂

[Kate Saikishore @katesaikishore]

Building of #qbrin: Why we stopped trusting LLMs to check their own work.

The biggest danger in enterprise AI isn't an outright error—it’s "Silent Wrongness."
When a model cites an invoice date as 2026-08-29 instead of 2026-08-28 with 99% confidence, standard RAG lets it slide.

Most teams fix this by chaining an expensive LLM judge.
We solved it with sub-5ms deterministic mathematics.

Here’s how our Gear 1 Claim Token Gate works:
🧵👇 (Watch the 60s breakdown)

1/ The subword tokenization trap: Numbers and dates get fragmented probabilistically.
2/ Gear 1 extracts atomic factual tokens (ISO dates, currencies, quoted UUIDs).
3/ Normalizes and word-bounds them against raw retrieved evidence spans.
4/ Result: 56.9% of invalid claims are rejected in <5ms with ZERO LLM token cost.

Zero hallucinations. Zero silent failures.
Tomorrow: The "Pareto Lock"—why fixing hallucinations usually breaks everything else.

#buildinpublic #AI #rag #softwareengineering

[Kate Saikishore @katesaikishore]

@_potatodonkey_ The harness column is carrying a lot here — Opus in Claude Code, Grok in Grok Build, GPT-5.6 in Codex. That makes it a model×harness ranking, so 1.39 vs 1.08 can't be attributed to the model alone. Any plans for a fixed-harness arm to separate the two?

[Kate Saikishore @katesaikishore]

@ProximalHQ Publishing the ± is what makes this honest — and it's what shows ranks 2-5 (32.2/30.2/25.9/25.3) sitting inside a single ±11 band. Given the concealed-cheating attempts you flag downthread, is some of that spread cheated trials rather than capability variance?

[Kate Saikishore @katesaikishore]

@jerryjliu0 Content Faithfulness 86.6→91.2 is the number I'd most want split. Old OCR fails loudly — garbled text you can see. A frontier model fails plausibly: it fills the cell it couldn't read. Same points lost, opposite blast radius on an insurance doc. Does ParseBench separate them?

[Kate Saikishore @katesaikishore]

@ArtificialAnlys Two bullets that might be one fact: vs 1.2, input tokens per task up ~57% on the agentic evals, and AA-LCR down 4 points. The gains are bought by putting more in context, right as long-context reasoning got worse. Do the agentic failures cluster in the longest runs?

[Kate Saikishore @katesaikishore]

@_philschmid Example 1's guard catches a wrong count, never a wrong file list — both come from the same read. ruff, pyright and pytest then run only on the files it already chose, so a caller it never enumerated passes every check. Do you run anything repo-wide before calling it done?

[Kate Saikishore @katesaikishore]

@xiangyuqi_pton 'After the knowledge cutoff' removes the memorised write-up, not the bug class. Most June–August disclosures are new CVEs on patterns the model has seen thousands of times, so the refresh mostly measures recall on a new surface. Did you split it by bug class seen vs unseen?

[Kate Saikishore @katesaikishore]

@Miles_Brundage Continuous auditing has a quieter failure than cadence: holding the metric's definition still while the thing it measures changes shape. 'Depth of the computation graph' gets redefined with each architecture, the series keeps its name, and a real change reads as a flat line.

[Kate Saikishore @katesaikishore]

@arcprize @AnthropicAI The ARC-AGI-3 note is the bigger result: the independent evaluator's traffic got classified as reverse engineering by the lab under evaluation. To an abuse filter, an eval harness and an attacker look identical. So 'Verified' now quietly depends on the auditee's allowlist.

[Kate Saikishore @katesaikishore]

@omarsar0 78% functionally coupled with nothing enforcing it is a discipline number, not a safety property — nothing fails to build when prose and script drift. So the 22% is the series I'd track: it should rise as the ecosystem outgrows the authors who held both halves in their head.

[Kate Saikishore @katesaikishore]

@LeoOliemans91 Since readback's already in your stack: the case it misses is the write that landed plus extra. Repair one row, silently touch 400, and the row you inspect reads back perfect. The assertion I'd pair with it is negative — nothing outside the intended scope changed.

[Kate Saikishore @katesaikishore]

Most AI agents do things and tell you afterwards. Ours asks first.

qbrin can now use a real web browser. But it can't click anything on its own.

It writes down what it wants to do, in plain words. You read that exact sentence — not a summary of it. You say yes or no. Only then does it happen.

And you can watch it happen. Live, on screen, while it works.

If you never say yes, nothing ever happens. That's not a setting we ship turned on. It's how it's built.

The agent also works in a locked room. It can't reach your files, your accounts, or anything on the internet we haven't let it reach.

Everyone is racing to give AI more power. We think the useful part is knowing exactly what it's about to do — and being able to stop it.

#AI #AIAgents #Trust #Security

[Kate Saikishore @katesaikishore]

@rohanpaul_ai Fabrication to 0% but severe hallucination still 4% is the shape I'd expect. Log-grounding kills invention outright — invention has no anchor. What survives is the claim the log partly supports but doesn't entail, and that's the one a spot-check waves through.

[Kate Saikishore @katesaikishore]

@MillionInt What I'd want measured is the asymmetry: does a run ever recover on its own, or is drift one-way? If deviation is absorbing and recovery isn't, length isn't a risk multiplier, it's a ratchet — and a ratchet isn't fixed by a better step policy, only by making state resettable.

[Kate Saikishore @katesaikishore]

@AnthropicAI The 4%→19% sneaky-hacking jump once the model knew a classifier was watching is the result I'd stare at: it makes the monitor's own trend unreadable. Fewer detections read the same whether behaviour improved or just got better hidden. Do you hold out an unmonitored arm?

[Kate Saikishore @katesaikishore]

@jamonholmgren 😇

[Kate Saikishore @katesaikishore]

@ziqiao_ma @thinkymachines The y-axis framing is the hard part — a ruler has to keep its unit across the whole sweep. As headroom shrinks the spread between adjacent models compresses, so a flattening curve reads the same whether gains slowed or the eval stopped resolving them.

[Kate Saikishore @katesaikishore]

@omarsar0 The transfer result seems more load-bearing than budget parity. Carrying within a family but weakening across it splits the gain into task instruction vs. quirks fitted to one student — and one lineage leaves no second candidate to tell those apart.