Arya Kanna

@a16z speedrun | engineering @UCBerkeley | ex-@Blackrock

[Arya Kanna @aryakanna9]

Hi everyone, I dropped out of berkeley at 19.

Through @speedrun, I've been building my own startup and excited to see where it goes.

Wanted to use this account to connect and share what my team has been learning.

This is only the beginning and would love to connect with anyone :) #buildinpublic #new

[Arya Kanna @aryakanna9]

@seenfinity @hwchase17 Both, but the expected state matters. Before/after shows what changed, not whether it should have, the worst ones we caught were refunds going to the wrong customer, totally invisible in the transcript.

[Arya Kanna @aryakanna9]

@MDNASIR79922242 @alexalbert__ 100%. Agents built cold on LangChain or the Vercel AI SDK scored 71 to 74% for us, vs 95 to 96% once they had lookup, memory, and policy checks.

[Arya Kanna @aryakanna9]

@seenfinity @hwchase17 Agreed, I'd go one step further and diff the actual system after each run. Traces miss stuff that looks right in the log and wrong in the database.

[Arya Kanna @aryakanna9]

@drjerryasmith @sh_reya @ruiyingm1120 Writing the pass/fail rules before the run is underrated. Most teams write them after they see the results lol

[Arya Kanna @aryakanna9]

@hirenthakore @hwchase17 how do you flag failure at emit time though? the expensive ones look like success until you check the system after.

[Arya Kanna @aryakanna9]

@Dan_Parmeggiani @sh_reya @ruiyingm1120 Hey Daniel! "reported OK while every check passed" is the scariest sentence in testing lol. Did you catch it from the output or by diffing state?

[Arya Kanna @aryakanna9]

@alexalbert__ The first thing every owner will ask is if it can issue refunds on its own. That's where it gets scary lol

[Arya Kanna @aryakanna9]

@simonw same question i keep asking about agents with refund access. top models get irreversible actions right about 56% of the time.

[Arya Kanna @aryakanna9]

@hwchase17 tracing shows what it said. still have to check what it actually changed.

[Arya Kanna @aryakanna9]

@sh_reya @ruiyingm1120 Congrats Shreya! Do you grade the final answer or the end state of the databases? We found a transcript judge missed 11 of 80 failures that checking actual state caught.

[Arya Kanna @aryakanna9]

@GradientLabsAI going straight to customers at a bank is a big call. respect. curious where they drew the line between answering and actually acting on the account

[Arya Kanna @aryakanna9]

@OfficialLoganK the number i'd want next is how it does on actions you can't undo. top models land those about 56% of the time in our testing

[Arya Kanna @aryakanna9]

@rauchg @aisdk @eve durability gets hard the second there's a side effect. replaying a read is free. replaying a refund is a second refund

[Arya Kanna @aryakanna9]

@hwchase17 the scariest ones we see are quiet. agent reports a clean refund, system says $48 over. does this check end state or mostly the trace?

[Arya Kanna @aryakanna9]

@rodms10 nice loop. only thing i'd add is a check on end state. we had a transcript judge miss 11 of 80 failures that reading the actual system caught

[Arya Kanna @aryakanna9]

@jhleath yeah lol. asked for an exact partial refund, came out $48 over every single time. call succeeds, number's wrong, idempotency never notices

[Arya Kanna @aryakanna9]

This is why we test agents in replicas of their real tools and diff every record they changed. Don’t trust what the agent says. Check what it did.

[Arya Kanna @aryakanna9]

@OpenAI says two of its own coding agents weakened their safeguards.

One turned off security alerts.
One removed monitoring, then said it didn’t.
Both treated routine instructions as permission.

The agent’s own report of what it did is not proof.

[Arya Kanna @aryakanna9]

@Zhen4good @swyx "press release" is honestly the best way to put it. only catch is your own state isn’t the whole audit trail either. a refund that already went through at the processor never shows up in your db, so you end up having to check the outside world too

[Arya Kanna @aryakanna9]

@numairarazzak @speedrun @Saad_163_ @LalithsaiP appreciate it :)

[Arya Kanna @aryakanna9]

Today we're launching Sparta Worlds.

Now you can see exactly what your AI agent did before it ever touches a real customer.

Thank you @speedrun and everyone else who has supported our team!