Kubernetes ships with self-healing. A controller holds the state you declared, compares it with what's running, and acts on the difference, so a crashed pod gets replaced and nobody gets paged.
My scheduled apps run on Fly, where nothing holds that state for them.
The same pass compares every app's secrets with its manifest, so a newly added client shows up as a missing database login, not a client the job quietly skips. And nothing deploys, automatically or by hand, while the test suite is red. About 1,400 tests have to pass first.
Policy-as-code separates where a rule is decided from where it's enforced: one decision point, and enforcement points that all read from it.
On my platform that rule is which models may do which work. Skills and scheduled jobs ask for a type of job, never a model.
Every route carries the date it was verified and warns after 120 days, and a regression runner re-scores each one over several trials on whatever model it points to now, so a quiet model swap shows up in my morning inbox.
Eight routes are cleared today, and one is on trial.
In infrastructure as code you don't hand-build an environment. You define a module once, create one copy per tenant, run a plan before anything exists, and accept the result only when a test passes.
New clients work the same way on my platform.
Once it's applied, the client registry and the engineer map regenerate, and a preflight connects to each database, checks the repo is visible and the signal key is live, and exits non-zero if any of it fails.
status says.
The checker runs on my laptop, so a job on Fly checks that the laptop's digest went out.
It's built for exactly the failure I saw in September, a scheduler that reported running for four days while running none of its jobs.
#PlatformEngineering#Observability#SRE
SRE has a rule for scheduled work: alert on the outcome, not the status. A process can report running all day and do nothing. The standard fix is a heartbeat, where the job reports that it ran and did real work, and an alarm fires when that report doesn't arrive.
Every scheduled job on my fleet runs through a small wrapper that writes a row to a shared table when it finishes. Each morning before the 9:30 digest, a checker compares about 40 jobs against their own schedules, so a job that was due at 7 with no row shows up whatever its
Dear GTM engineers: You'll probably have to spend some of your own money on tooling.
Paying out of pocket has been something I've done to give myself the extra bandwidth to build. Call it my R&D budget. I've been doing it for years. It's been part of what's gotten me here.