Jesse Chen

founding eng @usepylon, prev @samsara @cornell
my boss told me to get more followers, it's not going great.
exploring intersection of engineering , ai, product

[Jesse Chen @jessemchen]

A frontier AI lab gave us unlimited token usage for two months.

Our engineers already used AI throughout the day, often juggling anywhere from 3 to 20 agents at once.

We used the unlimited budget to push beyond everyday coding and test where else agents could accelerate the company.

Where possible, we gave them straightforward tasks with clear right and wrong answers. We also tested what happened when the work required more judgment.

Here’s what we tried, and how much value each experiment delivered:

5/10 - More PR review agents

We had multiple agents examine every PR from different angles. Useful, but our existing Codex review caught most of the same issues.

7/10 - Codebase sweeps

We added a `/sweep` command to search the entire codebase for bugs and performance improvements. It worked very well, especially when we pointed it at specific packages and gave it some direction.

8/10 - API documentation audit

Agents read our entire API documentation and codebase, identified mismatches between the docs and implementation, and cleaned up typos and spelling errors along the way.

2/10 - Linear triage

Agents reviewed every untriaged ticket and assessed whether it was actionable. I was really hoping this would create a self-improving loop, but either the Linear tickets were too vague or the correct action was too ambiguous.

3/10 - Trace optimization

Agents combed through our traces for slow queries, but struggled to close the loop consistently.

10/10 - Refactors

Agents finished all of our outstanding refactors that were mechanical or borderline mechanical. My trust in AI for this kind of work is nearly 100%. It is definitely more reliable than a human.

10/10 - Computer use for QA

After every significant refactor, agents open the app and test the affected flows like a user would. It has been reliable enough that I no longer have to manually QA every change.

The biggest lesson: unlimited tokens aren’t the bottleneck.

The docs match the code or they don’t. The tests pass or they don’t. The query gets faster or it doesn’t. The refactor preserves behavior or it doesn’t.

The moment the task required deciding what *should* be done, like triaging product tickets, performance fell apart.

That’s the biggest limitation of AI today: it can do an enormous amount of work, but I still can’t trust it to decide whether that work is right.

Unlimited tokens don’t solve the judgment and verification problem.

[Jesse Chen @jessemchen]

Google, Apple, and Meta can “just build it,” but they usually won’t. They have bigger fish to fry, they must chase big revenue with what they already have.

The startup playbook is to dominate a niche too small for them to care about, then use that wedge to expand into larger markets over and over again.

By the time they care, you’ve built something customers value that’s hard to replicate. That’s the moat.

[Jesse Chen @jessemchen]

@ChrisGPT Remember in 2018 when we thought self-driving was coming soon?
It's 2026 and I love Waymos but they are trying to get the costs and it's only in a few cities.
And automating white-collar labor >>> self driving in difficulty. Diffusion is always harder than you think it is

[Jesse Chen @jessemchen]

I can become 100x in that I can do 100x bug investigations with AI, and if the fix is simple for 10 of them, then I am a 10x bug fixer.

If I’m working on a larger project, I’m a 2x engineer. My agent takes a while to plan, run the tests, put up a PR, code review, etc

It’s really about working on things in parallel now. It will be way more exciting when a single thread speeds up as well.

[Jesse Chen @jessemchen]

Good take. A huge round gives you runway, but the valuation tells you what you’re allowed to become. You have to keep chasing enough ARR to justify the next round, which often means burning heavily on compute at the expense of durability and moat.

Earlier this year, new AI capabilities were being discovered every week, so investors chased anything that might capture them. Now we have a much clearer sense of where the technology can go, and the pace is starting to stabilize.

The labs going public will mark the next phase of the hype cycle: less speculation about capabilities and more scrutiny of the actual economics.

Some companies were priced as if the frenzy would continue forever. They’re going to have a rough time. But if you survive the cycle with a real product and durable growth, you win.

[Jesse Chen @jessemchen]

Most of these were true before the AI era other than “Build for agents, not humans” tbf

Build things that
1. provide value to customers
2. are hard for customers to build
So much of SaaS is reasoning through business problems, edge cases, change management.

I agree you are in trouble if an engineer can encapsulate all the edge cases of your SaaS in a few prompts and therefore can vibecode it.

[Jesse Chen @jessemchen]

yes exactly, it will be bounded by physical world (compute, literal time to complete experiments). So much time in AI research is waiting for things to complete.
I can buy RSI has already begun but theres a long way to go in solving other problems to unlock it.

Labs were hoping that AI would help them optimize at the model layer, but judging from the way they are experiment with swarms of agents, it feels like that’s the main way to solve difficult problems, feels more like brute forcing something verifiable.

[Jesse Chen @jessemchen]

Yep and the plateau is likely because the next level of coding is no longer as objectively correct/incorrect. Models never get logic wrong now. We are entering the “taste” territory which seems very difficult to train for. Or maybe context window is too small to fit the higher level picture into it, who knows

[Jesse Chen @jessemchen]

- Devin for 1st line of defense for bugs, alerts, design nits
- Conductor / Codex. No more TUI for me
- Codex/Cursor Bugbot for code review

Check out Conductor, it’s a visual wrapper around Codex/CC so you can launch agents in both from the same UI. Codex UI was inspired by them iirc

[Jesse Chen @jessemchen]

Eng work has accelerated so much that the dynamic between product + design has changed, since AI hasn’t sped up their work as much yet. PDE still needs alignment through on what to build which becomes the next bottleneck

Eng can parallelize work with many agents, but finding actionable work to use up all the agents can be hard. We try to fill it with bugs, performance work, security, tech debt because it doesn’t need input from outside of eng

[Jesse Chen @jessemchen]

@GergelyOrosz Astra/Fable for planning, Opus 5/5.6 Sol for implementation, Astra/Fable for review

If the change is risky, we run both Astra and Fable because they oftentimes find different things

[Jesse Chen @jessemchen]

@mitsuhiko The balance between moving fast and maintaining a strong mental model over the codebase.
What parts can become a black box for the sake of speed, and when should I continue to understand deeply to maintain understanding and control over the codebase as a whole

[Jesse Chen @jessemchen]

@chamath Idk about top 3 models being open source yet, but the gap is shrinking

Those companies will be winners though, their position is so flexible. The demand for their compute is already so high they can't serve everyone. And the demand will only go up, exponentially

[Jesse Chen @jessemchen]

Yep, and to add:
3. Too high ROI for the labs to monetize enterprise at the scale they need for their valuation, so it's hard for them to move up to the app layer where the stickier money is. They are spray + praying with law + biomed etc

For fun though, bull case:
1. Incredible talent density is a moat itself
2. They have enough talent + capital to acquire TypeSafe's or build their own Jev's
3. VCs + investors like Nvidia/Google/Amazon have significant stake, so they can manifest their success somehow

[Jesse Chen @jessemchen]

We’re still in the first inning of AI, yet valuations are already crowning permanent winners at the model layer.
Don’t sleep on Google, Amazon, or Apple. They control the stack from application distribution all the way down to hardware. That’s an incredible position if models commoditize.

[Jesse Chen @jessemchen]

@thorstenball Also “never surprised by a failing test” is not my experience lol
But I work in a large codebase with lots of complex business logic, so we might just be working on different stuff.
And I’m using Astra + Fable only

[Jesse Chen @jessemchen]

@thorstenball Oh I mean the agent writes unit tests, not hand-writing if that’s the misunderstanding

The agent uses it as some sort of “‘memory” of what the contract should be for something. That’s still valuable

[Jesse Chen @jessemchen]

@saranormous Econ is not my expertise but don’t you hire until the marginal utility of hiring the next person goes to zero, and AI makes everyone’s output 10x, thus more hiring?
Unless your company runs out of things to do, which seems unrealistic.
So it makes sense to me

[Jesse Chen @jessemchen]

1. RSI has yet to be proven
2. Even with RSI, talent would increase the rate of RSI
3. If you believe in RSI, the thesis is frontier labs should win bc their 6 month lead time snowballs to 2 years, 4 years, into ASI, and theoretically app layer gets eaten by all powerful models layer

I don’t think RSI is realistic even if we had infinite compute, and we don’t

But application layer is a good investment because that’s where most of the long term AI value is, but only if you think it won’t be eaten up by the model layer. Frontier labs are looking for the highest ROI app layer plays, so you just want to steer clear of those.

[Jesse Chen @jessemchen]

Exactly, if you’re not throwing AI at finding vulns in your software, you are asking to get hacked. There is no excuse - your AI has advantage of all the source code to find the holes. Invest in hooks to run security scans as you develop. Find all the recent hacks and have your agent ensure you don’t have similar weaknesses.