Ben Holmes

The Whiteboard Guy and Factory Foreman @warpdotdev

[Ben Holmes @BHolmesDev]

We built a skill for addressing the code review bottleneck: /merge-risk

This assesses the risk of any PR across 5 criteria: alignment, impact, verification, rollback cost, and error monitoring.

1. Alignment: Does the code follow the issue and PR description, or are there drive-by fixes and unintended side effects?
2. Impact: What is the blast radius of the change, and is it touching sensitive code paths like auth, permissions, or billing?
3. Verification: Is the code well verified, both with test coverage and end-to-end computer use artifacts?
4. Rollback cost: How easily can this be rolled back, and what is the rollback procedure in the case of an incident? Think one-way vs. two-way doors.
5. Error detection: If something goes wrong in production, how much visibility will we have? Are errors swallowed, logged, or reported to an error monitoring service?

You can run it as a PR comment automation, or make it a pre-PR check for the agent to derisk its own work before tagging in a human.

[Ben Holmes @BHolmesDev]

Get code review agents off of GitHub! I don't want to see the noise in my PR description.

Agents should run through their own internal code review process before a PR ever reaches a teammate.

[Ben Holmes @BHolmesDev]

@szavinny @warpdotdev Ah yeah, this conflict can be annoying. I'd disable the hidden top bar if you're a MacOS fullscreen user

[sathvik @vikvang1]

Catch us at AIE code summit in SF @warpdotdev

[Warp @warpdotdev]

You can bring your ChatGPT subscription to the Warp terminal, now with support for business plans too!

[Ben Holmes @BHolmesDev]

Use ChatGPT subs in the Warp terminal!

[OpenAI Developers @OpenAIDevs]

Now, you can use your ChatGPT Business plan in tools like @NotionHQ, @v0, @ampcode, @kilocode, @warpdotdev, Hermes from @NousResearch, @openclaw, @opencode, and more.

Connect with Sign in with ChatGPT to get started.

[Zach Lloyd @zachlloydtweets]

Lots of discussion about what a "software factory" is.

Also useful to call out what a software factory is not:
1/ A loose collection of automations around cloud agents
2/ A team of "AI Engineers" that developers manage interactively
3/ A set of skills and MCPs that individual engineers all adopt across a team
4/ A personal agent swarm

These are all useful things, but they aren't the right primitives for engineering scalable agent systems across an organization.

The right primitives allow you to define all your agentic workflows in code, run them entirely in the cloud, and, crucially, measure, test and improve those workflows over time to maximize agent productivity while minimizing cost.

If you start thinking from first principles "what is the system we should build as a company to get the most out of coding agents" you will end up with something like this.

[Ben Holmes @BHolmesDev]

I'll be live in just a moment to talk code review bottlenecks! We'll break down:

- Code review loops and how to benchmark their usefulness
- End-to-end verification skills
- A /merge-risk skill to mitigate risk ahead of PR

RSVP to join: https://t.co/3c2VniJoYS

[Ben Holmes @BHolmesDev]

Alright, it's time to address the code review bottleneck.

Here's 3 steps worth adding to your agent workflows:
1. Build your own code review loop
2. Verify flows end-to-end with simulated environments
3. Score and mitigate risk with /merge-risk https://t.co/eZcdBfFqaK

[Zach Lloyd @zachlloydtweets]

My @aiDotEngineer NYC talk is now on YouTube. I explain what a software factory really is and how teams should think about building them:
https://t.co/0siUXIW2Tr

[Eleanor Berger @intellectronica]

This skill is really fun and useful. Excellent way to learn things with an agent. https://t.co/WcnjizS0zz

[Ben Holmes @BHolmesDev]

I was talking to an engineer the other day about this question:

How do you make a codebase easy-to-navigate by both agents and humans, when everything from the hello world is agent-built?

The answer is system architecture. Agents (like humans) copy bad patterns like a virus. It's a ball of clay that needs constant reshaping.

You can write down best practices into your factory's skills, yes, but keeping the codebase itself on the right path is the real goal.

It's too expensive to have an agent review the universe constantly, spitting out architecture change PRs no one wants to look at. But, say, a weekly automation with a frontier reasoning model (Fable 5.2 is a good one to try) that pulls out one improvement to your system architecture is a start. I like @steveruizok's prompt to "look for implicit state machines."

Will experiment with a factory automation like this and see how it goes.

[Warp @warpdotdev]

Fable 5.2 is a great systems thinker.

Trying using it to find implicit state machines in your apps to make your codebase easier for agents to navigate.

Live in Warp Factories + the Warp terminal

[Ben Holmes @BHolmesDev]

"We're not working on the product, we're working on the factory that builds the product."

"We're not product engineers, we're factory engineers."

[Matt Pocock @mattpocockuk]

Just wanted to defend the term "Software Factory" for a second.

First, the caveats:

It's bullshit. It sounds fancier than it really is.

It's a bandwagon. People are using it to get you and their VC's hyped.

But it's also referring to a real change.

Earlier this year, most SOTA teams were using agents to work on the product. Shipping features, fixing bugs.

But now, they're building automations to help agents work on the product for them. They've moved up a layer.

What do we call the thing they're building? Loops? Workflows? Some other generic label?

"Software Factory" is as good a term as any other.

- "We're not working on the product, we're working on the factory that builds the product."
- "We're not product engineers, we're factory engineers."

I like it. I'll continue using it.

[Ben Holmes @BHolmesDev]

@thdxr [unexpected end of stream]

[Ben Holmes @BHolmesDev]

@diegoarias941 @thdxr Reasoning…

[Ben Holmes @BHolmesDev]

@garychanhk825 @zachlloydtweets Yes, we're working on the right default template to start from! We're thinking of agent factories like codebase dependencies: start with a default, extend with behavior you want

[Ben Holmes @BHolmesDev]

@Vantaggio_ar @vikvang1 @warpdotdev Understood. That's why we've been building a benchmarking tool for everyone to measure their setup, instead of just running benchmarks on our data and hoping they generalize.

This is what we've published so far, but we'll share more findings soon: https://t.co/OEArHrLY6K

[Ben Holmes @BHolmesDev]

@termleech @warpdotdev Glad you're liking it! 🙌

[Chuck Haines @termleech]

Exactly why we are evaluating @warpdotdev right now and likely will adopt it wholesale (terminal and factory). Gives us the whole package.

[Ben Holmes @BHolmesDev]

@peterpme It points to his /pr skill, which this pulls inspiration from for "blast radius" and "two way doors." Thought these ideas deserved credit!

[Ben Holmes @BHolmesDev]

We built a skill for addressing the code review bottleneck: /merge-risk

This assesses the risk of any PR across 5 criteria: alignment, impact, verification, rollback cost, and error monitoring.

1. Alignment: Does the code follow the issue and PR description, or are there drive-by fixes and unintended side effects?
2. Impact: What is the blast radius of the change, and is it touching sensitive code paths like auth, permissions, or billing?
3. Verification: Is the code well verified, both with test coverage and end-to-end computer use artifacts?
4. Rollback cost: How easily can this be rolled back, and what is the rollback procedure in the case of an incident? Think one-way vs. two-way doors.
5. Error detection: If something goes wrong in production, how much visibility will we have? Are errors swallowed, logged, or reported to an error monitoring service?

You can run it as a PR comment automation, or make it a pre-PR check for the agent to derisk its own work before tagging in a human.

[Ben Holmes @BHolmesDev]

@LaneHenslee Attach this to codeowners and we have a winner 👏

[Ben Holmes @BHolmesDev]

My talk on self-improving agents is live on @aiDotEngineer!

I show how you can build a skill + a self improvement loop to have agents review their own work and improve their setup overtime.

Very relevant in a software factory context too

https://t.co/6TyE0SnwmQ