Habibur Rahman Shihab

Software engineer building products that solve real problems
Learning, creating, and improving every day

[Habibur Rahman Shihab @hrshihab_]

Empty retrieval beats a missing reranker.

If top-k is empty (or below a score floor), fail closed: refuse or say “insufficient evidence.”

Forcing weak chunks = polite hallucination. Swap models without that gate and you just burn money on the same bug.

[Habibur Rahman Shihab @hrshihab_]

Claude outage lesson: don't just retry the primary until users quit.

Fail-closed routing:
• probe → circuit break
• secondary / retrieval-only / human
• no silent mid-task model swap

Byte-stable prompts + prompt-cache beat panic swaps.

[Habibur Rahman Shihab @hrshihab_]

16 PRs from Lean+Opus is impressive — and incomplete without human-owned invariants.

A proof only says code matches the spec. If nobody on the team can challenge the specs, you just automated confident wrongness.

Keep: LLM proposes, Lean refuses, humans own the invariant list.

[Boris Cherny @bcherny]

I used Opus 5.5 to formally verify the Claude Agent SDK using Lean. A couple short prompts = 16 PRs fixing various bugs and race conditions. Video attached.

TLA+ also works well. I sometimes combine Lean and TLA+ to look for issues around data flow, concurrency, and state mgmt.

I don't know either language well, but Claude is excellent at both. This approach is super useful for formally modeling your code and finding bugs that a human probably wouldn't have spotted.

Is formal verification the future of coding (or at least, bug finding)?

[Habibur Rahman Shihab @hrshihab_]

@0xJ4yD3v Environment block re-inserted before every user message is a classic silent cache killer. Byte-stable prefix or nothing — I treat any mid-turn system mutation as a regression in the harness CI.

[Habibur Rahman Shihab @hrshihab_]

@ThomAub Formal verify as a routine pass only works if the team can read the specs. Otherwise you get 16 green PRs nobody can challenge. Pair Lean with a human-owned invariant list checked in next to the SDK.

[Habibur Rahman Shihab @hrshihab_]

@jatingargiitk Harness-level routing without telling the user is the scary part. If a gateway can silently relay to another model family, your evals and data policies are lying. Fail closed: pin model + provider in the request receipt.

[Habibur Rahman Shihab @hrshihab_]

@dzerb1 This is why prompt-cache + retrieval discipline beat model shopping. 120k in / 4k out means most of the bill is context you already paid to assemble — stabilize the prefix before you chase a cheaper decoder.

[Habibur Rahman Shihab @hrshihab_]

@Bull_lion_aire Sandbox-first is right as a training wheel. The graduation criteria I'd use: typed tool allowlist, network deny-by-default, and a human gate on any FS write outside the worktree — then leave the harness.

[Habibur Rahman Shihab @hrshihab_]

@shadowaguy The methodological point is the real story: LLM proposes candidates, Lean/TLA refuse the ones that violate invariants. Neither alone catches race conditions as cleanly as the pair — especially on retry/stream state machines.

[Habibur Rahman Shihab @hrshihab_]

@LuisKisters Same fear — enjoy the window, but pin your harness evals now. If a nerf lands, you want a frozen task set that proves the drop, not vibes from yesterday's chat.

[Habibur Rahman Shihab @hrshihab_]

@KaiChandra @AnthropicAI Silent canary formatting is a trust hit even when the model is better. If Fable was already echoing Opus 5.5 style mid-week, teams need a model-id in the response metadata — otherwise evals get attributed to the wrong weight.

[Habibur Rahman Shihab @hrshihab_]

@stack3mpty Per-branch Worker env is exactly the missing piece for agent loops — a verifiable deploy target beats another sandbox claim. Stable URL + isolated Durable Objects means the agent can fail a check without poisoning prod state.

[Habibur Rahman Shihab @hrshihab_]

@dannyintheloop Human gate still on is the adult move. Cleanup agents are great at volume; they are also great at confidently rewriting the wrong abstraction. I keep write tools behind an approve step until the diff is under a file budget.

[Habibur Rahman Shihab @hrshihab_]

@PA91978375 TTL countdown in the status line is underrated DX. Once people see '12m left' they stop treating a long afk as free. Pair it with a soft warn before the next message if the last turn was >45m idle.

[Habibur Rahman Shihab @hrshihab_]

@abhishekray The logging-proxy approach is the right debug tool here — cache hit/miss is otherwise invisible. Curious whether Opus 5.5 keeps the prefix hash stable across effort switches only for the same tool schema, or if mid-session tool adds still force a rewrite.

[Habibur Rahman Shihab @hrshihab_]

@RolloLucas That +10pts tracks with what I see when schema noise drops. Curious which fields you deleted first — duplicate IDs, verbose descriptions, or enum sprawl? Those three usually move accuracy more than shrinking the system prompt.

[Habibur Rahman Shihab @hrshihab_]

@ravsau I’d still strip filler/noise before the cuts step even if retention is the guide — raw transcript + retention finds the peaks, but cleaning first stops the skill from optimizing around ums, ads, and false chapter breaks.

[Habibur Rahman Shihab @hrshihab_]

@nonstopmatt One practical gate: run install/shell steps in a disposable sandbox first, and require a human click before any command that can rewrite the agent’s own prompt/config files. Non-technical is fine — the gate is the skill.

[Habibur Rahman Shihab @hrshihab_]

@smc_shaun @typesafeai If Gradeline’s judge layer ever starts soft-scoring into the answer prompt, pull that out — keep it as a hard escalate. The Civil 3D headless path stays cleaner when the LLM only sees cases that already passed the gate.

[Habibur Rahman Shihab @hrshihab_]

@Yasho_27_ Before you change the %, freeze a small bridging-question set and measure hit@k on those only. If sibling chunks already cover the bridge, raising overlap just burns tokens without moving retrieval.

[Habibur Rahman Shihab @hrshihab_]

@heykathan Then the next filter I use: if a logistic/GBM baseline already hits the SLA on labeled features, don’t escalate to an LLM. Keep the model for the residual cases where retrieval + reasoning actually moves precision.