MarcRod

Ops of a ~400k rides/mo platform with no 24/7 NOC. Building Stanch so the pager can act when nobody answers. The hand only knows how to stop.

[MarcRod @MarcosRGjr]

1/ The disk filled up on the box behind our driver leaderboard.

The service stayed "up". Dashboards green. But it had silently stopped
counting rides toward drivers' rankings. No error. No alert. For days.

That incident became a product. Today I open-sourced it.

[MarcRod @MarcosRGjr]

Today I open-sourced Stanch — a dead man's switch that acts.

Two incidents made it exist.

The first: an LLM agent in our support flow credited a driver's wallet
because it received a deposit receipt. It looked helpful. It moved real
money on a picture. That day, every sensitive decision in my stack —
money, auth, dispatch — became deterministic code. The LLM talks; it
never acts.

The second: the disk filled up on the box behind our driver leaderboard.
The service stayed "up". Dashboards green. But rides had silently stopped
counting toward drivers' rankings. No error. No alert. A business
invariant, broken in silence.

Stanch is those two lessons as code: an external watchdog with dead-man
heartbeats and business-invariant rules in YAML, alerts on
Telegram/WhatsApp — and if a critical alert gets no human acknowledgment
in time, it executes the most conservative pre-approved STOP action. It
can pause, disable, quarantine. It can never start, transfer or delete
anything, by construction.

Apache-2.0, 228 tests, and an honest README: it states plainly what has
and hasn't run in production. I'm rolling it out shadow-first on my own
operation (~400,000 rides/month) and will report the results publicly.

If you run systems too small for a NOC but too important to bleed
overnight, I'd genuinely value your critique:
https://t.co/zx7KGzZ4wT

[MarcRod @MarcosRGjr]

Solo-ish founder running a ride-hailing platform (~400k rides/mo, 5 cities,
no NOC). The box behind our driver leaderboard filled its disk and silently
stopped ingesting ride reports. Service "up", dashboards green, rides not
counting. No error surfaced anywhere — a business invariant broken in
silence until a human happened to look.

So I open-sourced Stanch: external dead-man heartbeats + business-invariant
rules in YAML ("payouts/hour must not exceed X"), alerts on Telegram/WhatsApp,
and the controversial part — if a P0 gets no human ACK within a window, it
runs the most conservative pre-approved stop action for that rule.

Design constraints I'd love critique on:

- The action set is closed and stop-only (pause / disable flag / read-only /
quarantine / restart-once). It cannot start, transfer, delete, or touch
DNS, by construction (TypeError at registration AND execution, plus a
negative test that tries to sneak a rogue action in). Is that the right
line?
- Enforcement is opt-in per rule, shadow-first by default. Would you ever
turn it on, and for what class of invariant?
- ACK = ownership, not recovery: an acked alert stays active (holds the
dedupe fingerprint, auto-resolves on recovery). Does that match your
mental model of paging?
- Deliberately zero LLM in the decision loop (an LLM agent once credited a
driver's wallet off a deposit receipt in our support flow — never again
for money paths). Deterministic YAML only. Reasonable, or too rigid?

Apache-2.0, FastAPI+Postgres, 228 tests, live demo via make demo-incident.
https://t.co/zx7KGzZ4wT — brutal honesty welcome.

[MarcRod @MarcosRGjr]

7/ Apache-2.0 · FastAPI + Postgres · 228 tests
`make demo-incident` shows the whole loop live in your terminal —
and `make demo-ack` lets the human win the race.

Rolling it out shadow-first on my own ~400k rides/mo operation now.

https://t.co/3JMByYUh5O — tear the design apart.

[MarcRod @MarcosRGjr]

1/ The disk filled up on the box behind our driver leaderboard.

The service stayed "up". Dashboards green. But it had silently stopped
counting rides toward drivers' rankings. No error. No alert. For days.

That incident became a product. Today I open-sourced it.

[MarcRod @MarcosRGjr]

6/ The hand only knows how to stop.

Closed action set: pause, disable flag, read-only, quarantine,
restart-once. It can NEVER start, transfer, delete, or touch DNS —
enforced by type, proven by test.

Shadow-first. Opt-in per rule. A human ACK stands it down.

[MarcRod @MarcosRGjr]

Green dashboards lie. I open-sourced the watchdog that guards my own ride-hailing platform — and acts when nobody ACKs the page → https://t.co/3JMByYUh5O