https://t.co/hTexDZkoY7 is a great, objective, timeless way to evaluate models at identifying vulnerabilities. By evaluating models on vulnerabilities that occur after their training cutoff date, you can benchmark each of them against how many known vuln they discover. In my first tests, astra 6.2 actually achieved the lowest false-negative rate -- meaning where other models optimistically discarded files as "not vulnerable", astra 6.2 correctly identified them as "vulnerable".
Condition | Strict FNR | Including needs_context |
|---|---:|---:|
| GPT-6.2-Astra | 16.7% | 16.7% |
| Prior GPT-6-Astra | 69.4% | 38.9% |
| abliteration-large-v2 | 69.4% | 69.4% |
| Claude Opus 5.5 | 72.2% | 72.2% |
Lessons from building our own software factory
— AI still isn’t good at maintaining a modular, decoupled codebase
— When you loosen the reins, AI will create tables with unclear purpose & meaning, muddying your schemas
— It’s best to automatically generate documentation as part of your build/ci process. This way it’s deterministic.
— Simple orchestration of graphs and loops wins out over ambitious systematization of all processes.
— Attribution is tricky. When an engineer requests a change from the factory, who should be responsible for it? The factory, or the engineer?
— Designing upfront is still necessary. If you only program with goals (“add this integration”) the model will still shortcut to unmaintainable or inextensible solutions. This leads to cruft that has to be constantly reworked until you throw it away and design from the ground up.
— Maintaining control through feedback is important. That means reviews are as important as ever. It doesn’t mean reviewing the code. It does mean reviewing the architecture of what changed, and the durable artifacts (i.e. tables, packages) the change modified.
— A/B test your models. Give similar models the same tasks and see which one solves it better. Do this again and build trust. Benchmarks mean less and taste matters more.
— Having an agent produce screen recordings? Good start. Even better is giving the user a live VM where they can view the changes /in vitro/.
— Security is more important than ever. Every change needs to be redteamed by abliterated models. Especially while Anthropic and others gatekeep access to this ability. No exceptions.
What have you learned building your software factory?
Hot take: All coding is vibe coding now. As long as you're writing with specs and good principals, who cares if AI assisted in writing it? The people having breakdowns about this are going to have a rude awakening in a year when they find out all OSS projects use it.
We should not be building things that we don't understand.
Now the question is how to gain understanding over the things we're building. What are the techniques that we need to use in order to gain a deeper understanding of the code we're building, faster?
The funny thing is, they're the same techniques that we've always used. ORDs. Tables with comments. Architecture diagrams. Package structure. How do certain files talk to each other, and how tightly coupled are they?
I put together a little toolbox to help answer these questions. Using ladybug DB, treesitter, and others it builds understanding about your codebase and and tells you where the hotspots are, dead code, and others. Inspired by the pstack to help us try to wrangle these code cows 🤠 🐄
https://t.co/IYjUUdOE4Z
@poteto I love leaning into the idea of graphs and loops, especially dynamic ones, but the concept of perfectly specifying software upfront is impossible.
Instead I want to see people focus on A/B testing, produce multiple iterations of the same thing and compare them side-by-side.
Writing a scripting language that has filesystem as a first-class language feature.. Filesystem branches on parallels + checking out a previous state is dead simple. 🦉