Kris Tan

Working on better memory for AI at @Memoket_AI.

[Kris Tan @memokris]

Put the same sentence in two paragraphs, then ask for a replacement. A first-match search can make a perfectly reasonable edit in the wrong place. Our revision check rejects an ambiguous replacement unless the target span is bounded.

[Kris Tan @memokris]

Suppose a launch moves from June to July. "When do we launch?" needs the updated plan. "How did the plan change?" needs both notes. How do you mark the June record as superseded without hiding it from questions about the decision?

[Kris Tan @memokris]

Our client read the full response before passing chunks to the parser. The server was streaming normally. A slow synthetic stream makes the bug obvious: both paths produce the same sentence, but one consumer waits until the end.

[Kris Tan @memokris]

Two jobs read the same file, add different facts, and save. If saving replaces the whole file, both writes can succeed while one change disappears. Locking only the save is too late. The read and modification need to be inside the lock too

[Kris Tan @memokris]

An old edit may no longer fit after a manual change. In this prepared example, replay skips the change whose expected text no longer matches at its saved position. Other matching changes can still be applied.

[Kris Tan @memokris]

Take a source that says "target date, pending supplier confirmation." The draft keeps the date and says "delivery confirmed." The citation is relevant and the date is right, but the qualification is gone.

[Kris Tan @memokris]

They were enough to make us look past the final answer. The trace needed to show an actual source lookup, not just another plausible result from search.

[Kris Tan @memokris]

The agent needed two calls: find a fact, then use its ID to read the original text. Both tools were available, and the prompt described the sequence.

[Kris Tan @memokris]

We changed the loop to restrict broad searches in later iterations and leave room for calls that depend on an earlier result. Those four historical runs aren't enough to say how often this happens across models.

[Kris Tan @memokris]

A sentence in brackets described a chart that didn't exist. The AI still gave itself full marks. We rebuilt the case with sample inputs and illustrative scores, then changed a number in a chart that still rendered. The code checks caught both.

[Kris Tan @memokris]

If your retrieval eval only reports hit rate it's telling you almost nothing. Ours did, for weeks.

[Kris Tan @memokris]

Asked our AI memory what's my current job. It came back with one fact, about a project called Henderson. Completely unrelated. The right fact (joined as an AI engineer) was in the store. It just didn't come back.

[Kris Tan @memokris]

And the closer-to-real test, only the first 40% of a fact as the query (that's what writing looks like from the cursor): 15% to 88%. No model calls either way. A few milliseconds.