Om

20 | Building solo @refactron for the production codebase (

[Om @omsherikar]

Our own safety gate told us our flagship feature was unprovable 0 of 227 sites covered by tests.

So we killed the feature and pivoted the whole company around the gate.

Here's the story 🧵

[0xMarioNawfal @RoundtableSpace]

THREE CLAUDE PROMPTS YOU CAN USE TODAY:

1. “Act as a senior engineer. Think before coding, compare the best solutions, choose the strongest approach, write production-ready code, then review and improve your own work.”

2. ⁠”Turn every idea into a scalable AI business. Break it into steps, automate everything possible, identify bottlenecks, estimate cost, and create an execution plan.”

3. ⁠”Never agree by default. Challenge my assumptions, provide the strongest counterarguments, highlight risks and blind spots, then give your final recommendation.”

[Om @omsherikar]

@safishamsii How much?? maybe 20-30k??

[Om @omsherikar]

Yeah totally agreed!!

[Andrej Karpathy @karpathy]

We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". As one idea to generalize it, I was interested what Opus 5 would do if I gave it the first paragraph of the Lord of the Rings, a 1M token budget (~$10) and asked for three js render of it. Opus went off for ~2 hours and wrote 5500 lines of code that (procedurally) rendered the story. It's kind of janky but fun. But it's a bit mindboggling that the LLM has to place and orchestrate various polygon assets in (x,y,z) coordinates and write code that animates it all, and that it even does anything at all.

I also like this kind of examples because no one in their right mind would ever spend the time to write something this custom but LLMs have all the stamina and patience in the world, so it's an example where we go from "no one would ever do this" to "sure, why not, it's ~free". There might be a lot more. But I'm excited about creating hyper custom worlds that you can imagine dropping players into, e.g. here to participate in the LoTR story as a spectator NPC, or one of the characters, or etc. Something like an ephemeral GTA of X on demand.

Last thought is that the domain of worlds/games exposes a weakness in LLMs: they can't easily audit their work because they aren't able to efficiently and natively perceive videos or play games within them. Here, Opus 5 had to very slowly and painstakingly take screenshots at different points, and it messed up a few times and created a bunch of jank. An example of raw capability (multimodal, gameplay) that I think is still quite lacking.

[Om @omsherikar]

@mintlify It was the much needed thing for real

[Om @omsherikar]

@mintlify @refactron @handotdev @hahnbeelee I'm just in love with @mintlify how seamless it is!!

[Om @omsherikar]

Refactron × Mintlify 🔥

@refactron is now a part of Mintlify OSS.
Thanks, @mintlify for supporting us .
@handotdev @hahnbeelee

[Harshil Mathur @harshilmathur]

Someone’s vibe-coded “silly” app is another corporate security team’s worst nightmare :)

[Om @omsherikar]

am i that good? @AnthropicAI

#Anthropic #AFP #claude #Buildinpublic #devtools

[Om @omsherikar]

It's just insane right? made by texts from my portfolio.. just zoom it and see..

[Om @omsherikar]

@RoundtableSpace Just a marketing stunt

[Om @omsherikar]

@RoundtableSpace This is just impossible, made by editor or any other dedicated AI editor

[Om @omsherikar]

@ClaudeDevs did they really vibecoded this chart 53.4 % > 53.5% ??

[Om @omsherikar]

Excitedd to do some crazy stuff on this !!

[ClaudeDevs @ClaudeDevs]

Opus 5 is live in Claude Code and the Claude Platform today.

A few things worth knowing: 🧵
https://t.co/jAdUagXt9C

[Om @omsherikar]

Full write-up, including what this experiment does NOT prove: https://t.co/cPVa6H1u5M

Early access is open. Put the gate in front of your agent, over MCP, before its diffs land.

[Om @omsherikar]

Research on SWE-bench keeps finding the same thing: a big share of AI patches that pass every test are still wrong. One audit dropped a model's real solve rate from 12.47% to 3.97%. Green tests are not correctness. So we ran an experiment.

[Om @omsherikar]

The interesting one: the clean refactor of untested code came back UNPROVEN. Tests pass, but they never execute the change, so nobody has evidence either way.

A verdict that refuses to overclaim is the whole point.

[Om @omsherikar]

https://t.co/cPVa6H1u5M

[Om @omsherikar]

I asked an AI agent to plant disguised bugs in real codebases (Click, attrs), then verified its diffs blind with @refactron .
5/5 verdicts correct.
Both planted defects caught in under a minute, failing tests named.
The uncovered refactor: honestly UNPROVEN, not rubber-stamped.