devtocash

[devtocash @devtocash]

4/ One JSON ledger record per run, in S3 with Object Lock.

Payoff: a nightly diff of ledger vs audit logs. A write with no ledger entry = an agent outside its harness.

https://t.co/eIHdtDgLkC

#DevOps #Kubernetes

[devtocash @devtocash]

0/ Mint a ULID when the run starts, before the model sees a prompt.

Why not reuse the OTel trace ID? It's 32 hex chars nobody types into Athena at 03:00, and traces live 7 to 30 days while audit retention runs for years. Put the run ID on the span as an attribute instead.

[devtocash @devtocash]

3/ Git: commit trailers. Agent-Run, Approved-By, Prompt-SHA.

Searchable with git log --grep. What makes it a control instead of a convention: a required CI check on the bot's PRs that fails when a trailer is missing or Approved-By isn't a human.

[devtocash @devtocash]

Agent must open fix PRs? Run a second instance: own token, 3 tools (create_branch, push_files, create_pull_request). No merge.

Full setup 👇
https://t.co/IlvvJ1qtH1

#DevOps #AIAgents

[devtocash @devtocash]

Why it matters: in May 2025 Invariant Labs showed a "toxic agent flow".

Malicious issue in a public repo → agent reads it → pulls private repo data with the same token → publishes it in a PR.

No server bug needed. Untrusted input + private data + a write tool.

[devtocash @devtocash]

Log flooding: a failed matrix build is tens of MB.

Call get_job_logs with failed_only=true, return_content=true, tail_lines=200. Raise it (max 1000) only if the failing step isn't in the first 200.

Put that in the system prompt and verify it in evals.

[devtocash @devtocash]

Alert on it before a rollout surprises you:

(time() - kube_pod_deletion_timestamp) > 600

Full 5-step triage, plus the SIGTERM fix for pods that always take exactly 30s to die 👇
https://t.co/s9g6pQ7ZVs

#Kubernetes #SRE

[devtocash @devtocash]

First, the number that fools people: 30.

Default terminationGracePeriodSeconds is 30s. A pod 40 seconds into Terminating is draining, not stuck. Some workloads legitimately set 300 or 3600.

Only minutes past the grace period counts as stuck.

[devtocash @devtocash]

--grace-period=0 --force is the last resort, not the first. It deletes the object from etcd without anyone verifying the process is dead.

On a StatefulSet behind a partition, that's two pods with one identity writing the same data. Split-brain, by your own hand.