suleiman ribeiro

Building Gabby: a multi-tenant agent OS for SMBs. Solo, in production, since June. I publish what I measure — cost, latency, and the failures that stay green.

[suleiman ribeiro @suleimanribeiro]

I was pruning context to cut my agent's LLM bill.

Measured in production: $0.0168 per turn at a 9.6% cache hit rate. After the fix, $0.0036-0.0046 at 80-88%. Same model, same conversation.

-75% on the cached turn.

[suleiman ribeiro @suleimanribeiro]

"Whether it held up" is the part I spent the longest on.

Here the only signal that breaks the confidence ceiling is the revealed outcome — what actually happened afterwards. Approval moves the belief, feedback moves it, but neither reaches the top on its own.

How do you handle that one? Authority is still easier to model than whether something held.

[suleiman ribeiro @suleimanribeiro]

Everyone says the hard part of agent memory is deciding what to remember.

I tried to solve it with semantic similarity. Then I measured it, and killed it.

Three cosine scores from real production grounding:

format constraint: 0.588
brand color: 0.560
brand name: 0.667

[suleiman ribeiro @suleimanribeiro]

@stacktrace01 @hwchase17 Wrote up the rest of it — including why I killed cosine similarity as a credit signal, with the numbers.

https://t.co/j9ByoYsjQN

[suleiman ribeiro @suleimanribeiro]

Everyone says the hard part of agent memory is deciding what to remember.

I tried to solve it with semantic similarity. Then I measured it, and killed it.

Three cosine scores from real production grounding:

format constraint: 0.588
brand color: 0.560
brand name: 0.667

[suleiman ribeiro @suleimanribeiro]

Here it's by design, and the interesting part is the bug that almost broke it.

Validity is one thing only: a memory is alive while valid_until is null. Forgetting doesn't delete the row — it closes validity by touching that single field. History stays, recall only sees what's current, and because every reader goes through the same validity function, the cut applies to all of them at once. There are surfaces for it in the product: list, forget, and ask why the system believes something.

What almost broke it: one consumer filtered on a status column instead of real validity. That status was never written by any code path. Measured: 3 of 12 dead beliefs in dev, 59 in prod, being proposed as live.

That was the lesson. It isn't the delete that fails, it's the second consumer that asks the question differently. As long as "what is current" is a single function every reader must call, deletion works. The day someone writes their own filter, the dead fact comes back — and it comes back silently.

[suleiman ribeiro @suleimanribeiro]

The industry benchmarks memory on recall: did you retrieve the right chunk.

Inside a company that's the easy half.

The hard half is authority: who said it, what signal, when, and whether anything downstream ever proved them right.

[suleiman ribeiro @suleimanribeiro]

And the house style nobody writes down?

It isn't authored. It emerges from nightly consolidation, episodic to semantic, gated at two independent sources.

One mention is an anecdote. Two is a pattern.

[suleiman ribeiro @suleimanribeiro]

Related: "Don't Break the Cache" (arXiv 2601.06007) measured this across 500 agent sessions - 41-80% cost reduction from strategic cache boundaries.

My case was one layer up: the optimizer itself was invalidating the prefix, every single turn.

https://t.co/J0DHuDdKus

[suleiman ribeiro @suleimanribeiro]

I was pruning context to cut my agent's LLM bill.

Measured in production: $0.0168 per turn at a 9.6% cache hit rate. After the fix, $0.0036-0.0046 at 80-88%. Same model, same conversation.

-75% on the cached turn.

[suleiman ribeiro @suleimanribeiro]

These numbers come from a multi-tenant agent system I have been building solo since June, logged in a dated engineering diary. More findings as I measure them.