Engineer, Cornell MS. Built multi-provider LLM routing at PolyPoll. Building Cyan in public. Writing about AI agents and what actually works, with data.
I'm Ryan, a backend engineer. I build LLM systems and then check whether they actually work.
Some of what I've built: multi-provider LLM routing with failover at PolyPoll, a prediction market startup. SpanRedact, a small open-source tool that redacts common PII patterns from LLM trace spans before they're exported. And part-time at The Widget Factory, I review coding-agent runs and whether the tests grading them are fair.
I'm also building Cyan, where you set the boundaries and chance picks one thing inside them for you to go do. I recently tested how an AI assistant remembers a person, using synthetic users.
If you work on evals, agent reliability or AI memory, reply or DM me. I like comparing notes.
@calcsam Nice launch. When a request fails over, does the backup provider have to match on things like context length and quantization? Those can differ between hosts serving the same open model.
@giangnguyen2412 In the evals I review, the tests really are wrong sometimes, so a model calling out a test isn't a red flag by itself. In your setup, does a run that keeps the correct answer and flags the test count as a success?
@dexhorthy Drift is real, but tests only check what someone thought to test. On one coding-agent task I reviewed, every run passed its tests and got the same field type wrong, and only the spec showed it.
@dotgil The real one was a field's type. The spec was explicit about it, every run got it wrong the same way, and no test checked the type. A caller written from the spec wouldn't even compile.
Eval pattern #1: all tests pass, and it still doesn't match the spec
I review coding-agent runs part-time, mostly checking whether the tests grading them are fair. One review stuck with me. Every run passed every test, and every run got the same detail of the spec wrong. The tests never looked at that detail.
A made-up example with the same shape:
Spec: parse_date(s) returns a date object
Agent: returns the string "2026-11-15"
Test: str(parse_date("Nov 15, 2026")) == "2026-11-15"
Both versions pass. Then someone writes code from the spec, calls parse_date(s).year, and it breaks.
So I read passing runs against the spec too, line by line. If I'd only read the failures, I'd never have seen this one.
Someone asked me last week whether the write check in my AI memory experiment was throwing away facts that mattered. I didn't know. I wasn't logging what it dropped.
The check is simple. Every fact has to point to a real conversation event, or it doesn't get stored.
So I re-ran it over the saved extraction outputs. It had dropped nothing. Every source ID in roughly 32,000 writes matched a real event.
The losses were one step earlier. About a third of the wrong answers needed a fact that wasn't in memory, and every one of those facts had never been written at all.
A write-time check keeps junk out. It can't notice a fact that was never written. I'd still keep the check, but I'd log what it drops from day one.
(Synthetic users, one memory design.)
@sermakarevich Even specs that look complete run into this. In one coding-agent task I reviewed, the spec could be read two ways and the tests only accepted one. A run that picked the other reading would have lost points for a fair call.
@coldboothq Reading it. It only checked the test's own table, so it was visible right on the page. Breaking the code and watching it stay green would be the stronger proof.
@himanshutwtxs On the format drift paper, I hit a milder version in a memory eval. The grader marked correct answers wrong just for using a different word form, so it looked like the model was failing. Fixed it before the real run.
@TrailMarkin In the case I wrote about it was a type too. Tagging each test with the spec line it checks is a good idea, since the lines nobody tagged show you where to look first.
@asad0801 Here's a Door 2 case I found reviewing a coding-agent grader. In Go, the -run filter is a regex, so ^TestFoo also picks up a TestFooBar the agent wrote. If that one fails, a correct fix loses points, and ^TestFoo$ closes it.
@Prayag12B I've been building something close to this. Instead of a slider, you set your limits first and it picks one random thing inside them. My guess is people take a surprise better when they picked the range themselves.
@ez_ai4u Went back and counted. Every source ID matched a real event, so nothing got dropped in the full run. The misses that traced back to memory were facts that never got written in the first place.
@pauliusztin_ Same problem with generated data. In a memory test I ran on synthetic users, side characters' details drifted from one session to the next, so I had to check the data itself before trusting any score.
@stacknessdev@claudeai Most of mine end up in skills, and the ones I added later have the date next to them. One skill grew to 29KB before I cut it to 4.6KB by keeping only the rules and moving current status into memory.