ObserveCo

Founder @ ObserveCo. Humans can't see what happens under the AI hood. Silent failures, invisible costs, buggy systems. Built ObserveCo - See it. Fix it.

[ObserveCo @1571keplerj]

7 most common silent failure modes in production AI agents — and how I diagnose them in <5 minutes.

These patterns destroy reliability and waste money. I’ve seen them across Hermes, LangChain, custom fleets, etc.

Quick diagnostic checklist + fixes. No fluff.

#AIAgents

1/8

[ObserveCo @1571keplerj]

https://t.co/hBgQQTFqZ9 Every self-improvement loop has a hidden failure mode. The agent learns to satisfy the evaluator rather than genuinely improving. The moment the judge stops getting harder, the loop stalls and reward hacking creeps in.

The structural answer: Co-evolve the agent AND its evaluator together, so the bar keeps rising as the agent climbs.

[Miles Deutscher @milesdeutscher]

Anthropic's internal loop engineering playbook just got leaked.

And it's the most valuable AI guide I've read all year.

This guide is packed with info, and there are five things you need to know if you want to maximize your AI productivity with loops (save this):

1. You should structure every loop around these 5 principles:

• Discovery → Let the agent find its own work (CI fails, issues, commits)
• Handoff → Give every task its own isolated git worktree
• Verification → Never let the generator grade its own work
• Persistence → Always write state to disk (markdown or board)
• Scheduling → Run it on a timer so it works while you sleep

2. Separate Generator from Evaluator (most important rule)

Use two agents: one writes, the other is a skeptical judge who assumes the code is broken.

Make the evaluator act (run tests, click buttons, take screenshots) - this is what actually stops bad output.

3. Build with these 6 parts:

• Automations (the timer)
• Worktrees (safe parallelism)
• Skills (permanent project knowledge)
• Connectors (talk to GitHub, Linear, etc.)
• Sub-agents (generator + evaluator)
• Memory (state files that survive between runs)

4. Things you must watch out for with loop engineering:

• Verification debt (use verification agents)
• Losing understanding of your own codebase (start fresh if needed)
• Token costs exploding (solution below)
• Cognitive surrender (don't stop thinking because “the loop handles it”)

5. Solving token costs

Loop engineering can be crazy expensive.

I recommend you use an 80/20 "barbell" approach to loop engineering.

For your most complex tasks that require the best intelligence, use expensive models (Opus).

For the remaining 80% of your tasks (the gruntwork), use cheap, open-source models within the Claude Code harness (GLM-5.2 is great for code execution).

Save these 5 rules so you don't forget them.

[ObserveCo @1571keplerj]

https://t.co/GjyU8MvJfB

[ObserveCo @1571keplerj]

🧬 First open implementation of the Red Queen Gödel Machine

Self-improving agents eventually cheat. The evaluator goes stale, the agent learns to game it, the loop stalls.

The structural fix: co-evolve the evaluator alongside the agent.

Coding benchmarks - 1.35x–1.72x fewer tokens than prior SOTA

Scientific writing - 1.78x–1.86x higher acceptance rates

Proof grading - 9% higher ground-truth accuracy

Paper reviewing- Corrects 1.91x over-acceptance of AI-generated papers

Zero-dependency Python package. arXiv 2606.26294.

GitHub - observeco/rqgm-core · GitHub

@dair_ai @omarsar0 @Cambridge_Uni @NousResearch

[ObserveCo @1571keplerj]

First open implementation of the Red Queen Gödel Machine 🧬⚡

Co-evolving agents and evaluators for self-improving AI systems.

Fork of @SentientAGI's EvoSkill with epoch-based utility evolution: • Hack ratio detection → tolerances tighten when exploitation found • Adversarial scoring → penalises answers that game loose criteria • 35 unit tests, all passing

Based on arXiv 2606.26294 (Cambridge, June 2026)

https://t.co/TtMqnGjazk

[ObserveCo @1571keplerj]

First open implementation of the Red Queen Gödel Machine 🧬⚡

Co-evolving agents and evaluators for self-improving AI systems.

Fork of @SentientAGI's EvoSkill with epoch-based utility evolution: • Hack ratio detection → tolerances tighten when exploitation found • Adversarial scoring → penalises answers that game loose criteria • 35 unit tests, all passing

Based on arXiv 2606.26294 (Cambridge, June 2026)

https://t.co/TtMqnGjazk

@dair_ai @omarsar0 @SentientAGI

[Sentient @SentientAGI]

Sentient’s AI research team is paving the way for frontier labs like Alibaba Qwen, Google, and Microsoft through its contributions to self-evolving agents.

Here's a recap of how the biggest labs in AI are building on EvoSkill ↓

[ObserveCo @1571keplerj]

https://t.co/NTD1FITQrK

[ObserveCo @1571keplerj]

This is a really clean self-improvement loop. The combination of continuous triggers + periodic curator is one of the better designs I’ve seen for long-running agents. The missing piece most people underestimate is memory rot. Even with a curator, raw memory (and even agent-generated skills) tends to accumulate noise, contradictions, and low-value entries over weeks/months. Without active management, context quality degrades even if the agent keeps “learning.”

A few patterns that help significantly:
Salience / importance scoring: Instead of treating all memories equally, score entries on relevance, recency, usage frequency, and confidence. Low-salience items can be summarized, archived, or decayed (Ebbinghaus-style forgetting curve works surprisingly well here).

Periodic consolidation passes: Beyond skill curation, run regular memory consolidation jobs (e.g., nightly or weekly). These can merge related facts, resolve contradictions, compress verbose entries into summaries, and surface “stale but important” items that need human review.

Observability into memory state: The hardest part of long-running agents is knowing what the agent actually remembers vs what you think it remembers. Simple but powerful additions include:Ability to query memory by salience, age, or topic Visualization or summary stats (e.g., “Top 20% of memories account for 80% of retrievals”) Contradiction detection between new and existing entries before they get written

Without this layer, the self-improvement loop can quietly start reinforcing suboptimal or outdated knowledge.

[ObserveCo @1571keplerj]

This is one of the best threads on agent observability I’ve seen. The distinction you draw between service observability and agent observability is spot on — especially the non-determinism and delegation boundary problems.

A few patterns that have worked well for me when implementing traces/spans in agents:

1. Correlation ID propagation (lightweight version)
Instead of passing full metadata blocks in every prompt (which can bloat context), I generate a trace_id at task start and thread a compact correlation object through the tool dispatch layer only. The model never sees it unless I explicitly inject a summary for high-stakes decisions. This keeps prompts clean while still giving you a single causal tree across parent → child → grandchild delegation.

2. Span design for agents
I treat each turn as a parent span and each tool call as a child span. Key attributes I always include:
turn tool_name + toolset
inputs_hash (for quick diffing)
output_status (success / partial / error)
duration_ms
retry_count
context_delta_size (how much new context this turn added)

This makes it trivial to query “show me all turns where tool success rate dropped below 90%.”

3. Replay debugging that actually works
Your point about replay being magic is 100% correct. One refinement I’ve found useful:
instead of just replaying tool outputs, also capture and replay the exact model decision context (the prompt window + any retrieved memories/skills at that turn).
This lets you debug why the model chose a bad tool call in the first place, not just what happened after. The biggest win from structured traces isn’t just faster debugging — it’s turning “the agent did something weird” into reproducible, queryable events.

Once you have that, metrics like tool success rate and hallucination rate become leading indicators instead of lagging ones.

[ObserveCo @1571keplerj]

Once you treat Hermes as production infrastructure, a whole new class of problems appears. The deeper issue here is lack of observability into the agent's actual runtime state. When memory resets, skills fail silently, or the scheduler behaves unexpectedly, you often only find out after the fact (or when a human notices something is wrong). A few observability practices that help dramatically with these exact problems:

Pulse / heartbeat checks: Have the agent (or a lightweight wrapper) emit a regular signal (every 30–60 seconds) that includes current uptime, memory backend status, last successful skill load, and scheduler health. This turns "it just stopped working" into something you can alert on immediately.

Job / skill status tracking: Instead of relying only on Hermes logs, maintain an external lightweight status store (even a simple JSON or SQLite file) that records last execution time, success/failure, and output location for each scheduled skill. This makes silent failures visible without parsing every log line.

Absolute paths + monitoring: Always use absolute paths (as you mentioned) and monitor the actual filesystem state. A simple periodic check that verifies the expected output directory exists and has recent files catches many of these reset scenarios early.

Context & state snapshots on key events: Before/after major operations (memory writes, skill loads, restarts), snapshot relevant context or at least log a hash/summary. This makes it much easier to correlate "what the agent thought happened" vs "what actually persisted."

[ObserveCo @1571keplerj]

Many memory providers (or wrappers around them) return success on write acknowledgment rather than confirmed durability.

Agents treat the memory layer as a black box and only check high-level return values.

There’s usually no correlation between the write operation and later read operations in the same session/turn.

[ObserveCo @1571keplerj]

The pattern is always the same: you can’t fix what you can’t see quickly.

Observability (traces + metrics + memory inspection + health pulses) is now table stakes for serious agent work.

What’s the most painful silent failure you’ve hit lately?
Drop it below — happy to share more specific patterns.

[ObserveCo @1571keplerj]

7 most common silent failure modes in production AI agents — and how I diagnose them in <5 minutes.

These patterns destroy reliability and waste money. I’ve seen them across Hermes, LangChain, custom fleets, etc.

Quick diagnostic checklist + fixes. No fluff.

#AIAgents

1/8

[ObserveCo @1571keplerj]

#7 Scheduler / Cron Failures

Background jobs fail silently with no alert or retry visibility.

Diagnosis (<2 min):
Check job logs + last execution time vs expected schedule.

Fix: Add job-level observability, status tracking, and notifications on failure.

#LLM

8/8

[ObserveCo @1571keplerj]

Running Hermes 24/7 is next-level, but you need visibility into what’s actually happening under the hood. Ive created ObserveCo to add pulse monitoring, auto-recovery, error timelines, and memory analysis so your always-on agents don’t fail silently. Currently designed for Hermes. Would love feedback from people running long-lived setups.

How do you currently monitor your background agents?

[Lummox @Lummox_eth]

HERMES RUNS 24/7 IN THE CLOUD AND IMPROVES ITSELF EVERY 10 PROMPTS. HERE IS THE 3 PART LOOP.

Most agents forget everything when you close the tab.

Hermes runs continuously in Telegram, watches conversations, and writes what it learns into permanent memory and reusable skills automatically.

Trigger system fires every 10 user prompts to check for new facts, and every 10 tool-call iterations to detect complex problems worth turning into reusable skills.

A background agent runs in parallel updating MEMORY.md and creating skills tagged as agent-generated for full traceability.

Curator runs every 7 days during 2 hours of idle time.

Mechanical pass first : skills unused for 30 days deprecated, unused for 90 days archived.

Then LLM review decides to keep, fix, merge, or archive each skill with a full rename map documenting every decision.

Model routing sends auxiliary tasks independently : vision to Gemini, compression to local Qwen, curator to whatever you trust for long-term decisions.

No silent shuffling between 20 different quantizations.

Install is one command. Telegram setup requires no manual bot token.