We built a software dark factory to automate the development of an existing application. Last night it merged tens of PRs while we were asleep.
The following is a narration of our own experience applying the principles outlined in @_lopopolo's OpenAI post on harness engineering, which may well go down as one of the most influential software engineering posts in recent times.
New blog post on what would be true about the world if trendline continues and leading lab hits $1T in revenue by the end of next year.
In other words, why compute might get 10x+ more expensive in coming years
https://t.co/7PvniDC2jN
Thanks for sharing the experience. This helps.
In my case, I also had a version of my own 'software factory' but, since January I've been simplifying it more and more to be honest.
But you hit a critical difference. So I plan "by hand", and divide the work into max half-pager PR descriptions. (In the past these used to be extensive: I find that newer models get too stuck on large plans and its best to let them explore.) Sequential. So right now for instance, I'm working on a refactor in prep for a new feature, which is about 9 PRs. This I plan "manually" or with a lot of intervention. Even if there are parallel parts to it, I go sequential.
And of course I work on parallel domains/features but each is a sequence of PRs for simplicity. I have enough parallel/orthogonal work to keep me busy and quotas full as-is.
Then I have them implement it in sequence. There is a prompt checking and potentially halting for human-in-the-loop after each PR. Otherwise, the PRs get stacked together (stacked PRs).
I do have each PR reasonably small and I only ever use gpt 5.6 xhigh, because I simply don't "trust" anything else, unless it is truly trivial feature or fix. I use a combo of Fable + 5.6 Ultra for planning the PRs (but that's done "manually").
As for code, I follow very closely the "OpenAI Harness Engineering" blog code/packages structure; I think that helped us, but that's more for parallelization/organization of the domains to avoid constant conflicts.
I do sometimes use /goal for more autonomous work and experiments. That's a bit hit-or-miss. I still need to learn to 'abandon' failed /goal experiments rather than prompt them to a proper state. Frankly, I think 5.5 and earlier models did it a bit better there ironically; 5.6 needs more 'leeway' to explore and decide on-demand rather than follow a very strict, very detailed plan.
@alexelcu@fwbrasil So far, in my own work, I don’t find it very different than acting as an engineering manager. Mistakes pass through, we build more processes for async spot checks etc. I do find drastic differences in specific models and reasoning levels.
I just found out my kid is using a 8B parameter model that was fine-tuned to play Minecraft, just for fun (!) https://t.co/CXGeTxbQcM
I'm not sure which part of the hype cycle this is...
Day 3 of preparing a SaaS version of Kora. The ecosystem is genuinely great nowadays: Temporal Serverless workers, Vercel Sandboxes etc. And of course coding agents to build and manage autonomously long-running benchmarks to understand the behavior of these services.
@Jonathan_Blow@ErinJConnors Capabilities increase and pass new thresholds. I have various examples of projects that were unachievable without strict supervision and now work well. These systems are no longer stochastic parrots.
Tip of the day: having gpt 5.6 autonomously draft a few agreed upon design ideas, build and deploy them for real, purely as a test for benchmarking analysis and code reorganization, as part of the design/planning phase. Nothing like obtaining real data/numbers from real systems!
Ok, due to popular demand, I think it's time to build the SaaS/self-service version of Kora. Day 1, let's see how far we get... (It's not our 'first rodeo' building SaaS products, but ambition has grown!)
Day 2: Have an extensive plan (thank you 5.6 Ultra for the help) and 10+ PRs merged that refactor/simplify/remove services, as to accept new implementations, backends, etc.
I’m very pro-AI but fully agree there is space for deep artisan work, made with deep care and human touch. Retro gaming is one perfect example, where it should be the default but there are others, eg I also have no interest visiting a museum of AI paintings. And I’m happy to pay a significant premium for the human experience and deep work.
A few people were curious about what coding agents can build today and my comments on lower-level languages in the 'age of coding agents'. Well, here's a simple example: https://t.co/UdsU9FNyAx
Draft is a systems language, compiler and tiny IDE, built mostly by Codex over a few days. It is just an experiment on what's possible currently with some minimal guidance. It is not a product, nor a claim, nor any form of victory lap, but simply showcasing the current state for those curious.
I did steer the architecture as we went, especially compilation. When timings and larger programs exposed the first naive implementation based on serial Clang launches, repeated semantic work and overly broad emission, I pushed for a demand-driven semantic graph, deterministic parallel work, reachability-based emission and in-process LLVM. The agents then refactored toward that happily. So first indications is that it is very possible.
In case you cancelled Cursor but still use it as an IDE, this is worth knowing. It is unclear how much they use data for training vs plain indexing, but terms may change anyhow. Cursor at least offers a privacy mode, even on free accounts.
I don't pay for Cursor. My subscription has been cancelled for months.
But between July 18–27 it uploaded 63,106 files — ~736 MB of my source code — to Cursor's servers. (These include imported libraries etc as well by the way).
Just because of opening the folders, to use Cursor as a pure IDE. The worst part is that there is no way to delete it.
Why would you want my code uploaded if I'm not even using your AI services @cursor_ai?
Betting on custom AI silicon is betting model architectures won’t change. They will. Optimize for today’s model and tomorrow’s becomes a misfit. The old computing lesson: point designs win benchmarks; programmable platforms win the era. That means GPUs win, NVIDIA wins.