AI Ops Notes

Notes on AI workflows, agents, and operations.
Better systems over better prompts.

[AI Ops Notes @nssntus]

AI was supposed to make us work less.

Instead, 1 in 7 workers using AI say they've experienced "AI brain fry."

The bottleneck isn't compute anymore. It's your attention 👇 https://t.co/lyjEQnYJ4S

[AI Ops Notes @nssntus]

When you turn on the egress gateway, self-hosted Claude Managed Agents opens port 8410 on every interface, plain HTTP, and its answers carry stored credential values. It also keeps sessions in memory by default, so one restart erases every session. You get custody of agent history and credentials, and you inherit jobs the hosted service did for you.

My take: "keep secrets in-house" is the pitch, but the riskiest parts in the docs are internal ports, not model calls. That 8410 listener is for the egress gateway, and Anthropic's Compose and Helm setups lock it to the gateway; run it any other way and that's on you. Turn on Prometheus through the Helm chart and port 9464 asks for no credential. And one deployment is one trust level: the server API key reaches every environment.

Before you turn it on:
1. Configure a database, or accept losing sessions on restart.
2. Only the gateway reaches 8410. Firewall 9464.
3. Separate deployments for each tenant, and for trusted vs untrusted code.
4. Plan for what's missing: no webhooks, no scheduled runs (bring your own cron), and MCP OAuth isn't supported, so you store static bearer tokens and rotate them by hand.

Sources: Claude Platform release notes, Claude Managed Agents self-hosted docs

[AI Ops Notes @nssntus]

@expo @JI a recorded run before review is the right default, reading a diff blind is where reviewer time goes

[AI Ops Notes @nssntus]

@OfficialLoganK @GeminiApp @GoogleAIStudio @antigravity curious how it does as the executor under a bigger planner, that split is where costs drop

[AI Ops Notes @nssntus]

ok, Nvidia board is dumb

[AI Ops Notes @nssntus]

No company is dumb enough to fire the founder who led them to success over a sex scandal, of all things. My bet Nvidia won't touch Jensen Huang. Not unless the board has lost its mind, or there's something bigger we're not being told.

[AI Ops Notes @nssntus]

@emollick turning what a teacher says into something students can poke at is the part slides never did

[AI Ops Notes @nssntus]

A 2,000-token support ticket costs $0.0002 to check with Google's new Gemini Decisions API, whether you ask it 1 question or 30. The shared input is billed once per request (each question adds only its own instruction tokens). It doesn't write answers. It scores options: yes/no, pick 1 of up to 255, or grade on a rubric, up to 1,000 questions per request, each with a 0 to 1 confidence. $0.10 per 1M input tokens, zero output tokens billed.

My take: most agent stacks pay a generative model to do work that's really a multiple-choice question. Which tool, which model tier, is this safe, is this context relevant. Those should move to a scorer, and the big model should only see what the scorer isn't sure about.

But the confidence number is the product, and "lowest calibration error" is Google's claim on Google's benchmarks. Check it on yours before you trust a threshold.

How to try it:
1. Take the decisions your agent makes with LLM calls today (tool pick, model routing). Ask them as "choice" questions on the same input in one request, on 200 cases you've already labeled.
2. Bucket by confidence, find the cutoff where accuracy holds. Above it, act on the scorer. Below it, fall back to the big model or a human.

Sources: Google blog (Gemini Decisions API), Gemini API pricing page

[AI Ops Notes @nssntus]

@erden_xi probably yes, but not sure if I will be fired even not been paid. 😂

[AI Ops Notes @nssntus]

@elonmusk @zerohedge it would be great to see a company have Elon, Boris and Jensen works together.

[AI Ops Notes @nssntus]

@LDT0545 if unlimited, I would go any model, but still keep at least a $20 claude for advance planning work, and leave the work to the unlimited model.

[AI Ops Notes @nssntus]

@luisnhenrich with the beta header on, input_transformations lists each drop, log that next to the score too

[AI Ops Notes @nssntus]

Claude Fable 5.2 tops Google DeepMind and Artificial Analysis's new MKA benchmark at 57% (max effort), at $31.97 per task. Gemini 4 Carbon scores 54% for $6.29. Opus 5.5 scored 53% at max. That's Artificial Analysis's run: 200 private tasks where agents read drawings, schematics and clinical scans, then deliver reports, decks and spreadsheets.

The leaderboard #1 isn't the model you should default to. Anthropic's own docs say Fable's default effort is high, max is for measuring the ceiling, and most workloads should start with Opus. And routing bites quietly: if a router or fallback moves a conversation off Fable 5.2, to Opus 5.5 or even Fable 5.1, the thinking blocks are dropped and the request still succeeds. Fable's refusal fallbacks (Opus 4.8 and Opus 5) drop them too.

My take: "use the top model" is the wrong default for agents now. First and third on MKA are 4 points apart; Fable over Carbon is 3 points for about 5x the cost. Route by task, and pay for Fable only where you've measured that it wins.

Before you switch:
1. Run your evals on Opus at high or xhigh first. Send only the tasks it fails to Fable.
2. Sweep Fable's effort starting at high, not max.
3. If anything routes between models, log dropped reasoning: with the thinking-binding-controls beta header, the response lists each drop in input_transformations.

Sources: Artificial Analysis MKA results, Google DeepMind MKA announcement, Claude Fable 5.2 docs

[AI Ops Notes @nssntus]

@aksenHQ that's LATENT's count off Apiiro's data, not our test, plain passwords don't match the key patterns scanners hunt for

[AI Ops Notes @nssntus]

104,185 public copies of .claude/settings.local.json on GitHub. 1,122 of them held at least one likely live secret (Apiiro didn't test them).

That's from Apiiro's scan (collected Nov 1-3): 1,379 secrets in total, including 379 database passwords, plus another 734 that survived only in git history. The cause is Claude Code's "Yes, and don't ask again" button. It's a file write: if the approved command had a password in it, the password now sits in plain text in your project folder.

Two details make this worse than a normal leak:
- By LATENT's count from Apiiro's figures, gitleaks and TruffleHog together missed 1 in 5 of those files. A password after PGPASSWORD= or sshpass -p looks like an ordinary word to a scanner hunting for key formats.
- Claude Code keeps the file out of commits through your machine's global git excludes, not the repo's .gitignore. A new laptop, a container, a copied folder, or a file that's already tracked gets no protection.

Anthropic's Boris Cherny says the file is meant to stay local and recommends secret scanning whatever agent you use. Fair, but Apiiro found a Claude co-authored commit touching the file in about half the affected repos.

My take: the approval prompt has quietly become a secrets store. Grant "always allow" to patterns like npm run test *, never to literal commands, reference $DB_URL instead of pasting the password, and treat agent config folders like .env.

Check each repo you've run an agent in:
1. git ls-files .claude (if settings.local.json shows up, it's committed)
2. git log --all --oneline -- .claude/ (if any version held a secret, rotate it, even if the file is clean now)
3. Rotate the secrets first. Then git rm --cached the file and add it to the repo's own .gitignore

Sources: Apiiro report, Claude Code docs, LATENT analysis, Hacker News thread

[AI Ops Notes @nssntus]

@simonw so real map data is a way harder test than the pelican svg, the geometry has to actually line up

[AI Ops Notes @nssntus]

@_philschmid @ArtificialAnlys tbh a 3 point gap at a fifth of the cost per task, for batch work that's an easy pick

[AI Ops Notes @nssntus]

@omarsar0 tbh a thin harness only works if it still blocks risky actions and checks results outside the agent

[AI Ops Notes @nssntus]

@elonmusk l like model 3 more than Y 😂

[AI Ops Notes @nssntus]

Anyone feel the same as I do that Grok Bot becomes very passive today? @grok

[AI Ops Notes @nssntus]

@omarsar0 tbh more agents only helps if checking their work doesn't become the bigger job 🤣