Adam Wolff

Claude Code @AnthropicAI 🤖
Avid cook, dedicated snow person, yoga enthusiast

[Adam Wolff @dmwlff]

I ended my time at @Meta as a director.

But I started as an engineer on FB Chat.

Everything about it was broken — we had to rewrite it.

And while the effort to fix it is one the projects that led to @reactjs, the most important fix was far simpler...

Here’s the full story:

—

I worked on Facebook Chat for several years, both on the front end and the infrastructure.

Before the major effort to redo the UI, FB Chat was super broken and we had no idea why.

We got tons of bug reports about Chat being broken every day, but we noticed an odd pattern in the data: the volume of reports didn’t match the volume of usage. It was time-shifted from the peaks we’d see in the US.

We didn’t know what was wrong, but we knew the code was a mess.

We set about rewriting both the front-end and the back-end in an effort to fix it.

The front-end rewrite pulled in a whole team of amazing engineers and became one of the big threads that led to @reactjs

In the public eye, we portrayed this project as the one that ultimately fixed Chat.

And the way I’ve usually told it, fixing Facebook Chat and the birth of React are the same story.

But no framework was going to fix the worst problem with Chat.

—

During the time we were working on the Chat rewrite, we were also replacing the original Erlang backend with one written in C++.

This was probably a good move, but the problem wasn’t with Erlang either.

Our initial spec for the new backend didn’t say much about observability, but it was an important feature, and the rewrite forced us to rebuild it.

Little did we know this would lead us to the root cause of our problems…

When we finally gained insight into our deliverability data, we were able to cut it by region.

We noticed Chat was really popular in India. This was before WhatsApp, at a time when SMS wasn’t reliable.

Eventually we pinpointed a region in India where one specific DNS provider was giving out the wrong IP addresses for our Chat servers.

So when people went to use Chat, they would sometimes get a notification that they had a message, and then it would disappear.

Or they’d send a message and it would get lost. All because they were connecting to the wrong IP address.

That was it!

None of the sexy new tech we were working on was going to solve that problem.

Ever.

—

Instead, the solution was to build observability that allowed us to track end-to-end message delivery.

In the end, we could start with a broad cut of our data by country or web browser, and then zoom all the way in to look at what happened to a specific message for a specific user.

Once we pinpointed that the problem was with a DNS server, the matter was resolved with a quick phone call. I don’t know what they did, but I imagine it was something like turning it off and turning it on again.

We sometimes talk about observability as if it’s enough to buy a product like Datadog and just look at the pretty graphs.

Sure, that’s a start.

But true observability is a feature that needs to be built— painstakingly, iteratively, by-definition starting with a shot in the dark.

—

These days, it has become fashionable to poo-poo the idea of being data-driven.

People point out that measurement can distort the phenomenon that is being observed.

They want to make processes “data-informed.”

But this seems like silly backlash against the only rigorous standard in all of software engineering:

That we hold ourselves to an objective standard.

We measure how long things take, how many errors we encounter, how often a process successfully runs to completion.

So here’s what this experience taught me about observability:

When an issue happens in production, time-box the investigation.

Sure, take a few hours to try and figure it out by looking in the logs and inspecting the code.

But if you’re coming to the end of the day and you still don’t have a fix, then push a PR that adds logging.

The first one may be just a guess, but it will begin a process that leads to the truth.

And that is what we should all ultimately be striving for.
—
For more engineering tips and stories, follow me @dmwlff

[Adam Wolff @dmwlff]

/fork was sitting in the queue for a bit, but I've been using this feature for a while and finding it really helpful.

The best thing about fork is that you can get subagents to do a lot of work with small prompts since the context is shared. Bonus: the subagent reuses the prompt cache!

[ClaudeDevs @ClaudeDevs]

We've updated /fork in Claude Code

/fork now runs a background agent with your exact context (system prompt, tools, history, model) and prompt cache. The result gets returned to your session.

/branch (the old /fork) still copies the transcript to a new session you drive.

[Adam Wolff @dmwlff]

@gus_aragon @Neotk @trq212 this was a bad interaction between Claude Code and the new model. should be fixed if you update.

[Adam Wolff @dmwlff]

This whole thread is great, but this is the headline.

[Simon Last @simonlast]

2/ Think bigger. This is the most common mistake I see: tasks scoped too small. At this point you want to be aiming for work that would take a good engineer multiple weeks.

[Adam Wolff @dmwlff]

@Lon @xen_studio @trq212 We are working on refining the controls for this. For now, you can opt out by setting CLAUDE_CODE_FORK_SUBAGENT=0

[Adam Wolff @dmwlff]

@Lon @xen_studio @trq212 Thank you for this feedback. While forked subagents are mostly useful, there are definitely cases (like adversarial review) where you don't want it. You should be able to prompt Claude to use a non-forked subagent.

[Adam Wolff @dmwlff]

That is not accurate. Background/foreground does not affect cache at all. Forked agents (which are coupled with this experiment) should be _more_ cache efficient because they share cache with the parent and require less context to be passed as output tokens from the parent and unique input tokens to the subagent.

[Adam Wolff @dmwlff]

@Lon @xen_studio @trq212 The efficiency problem was fixed. Let me investigate the issue with permissions.

[Adam Wolff @dmwlff]

It seems like a century ago in AI time, but last fall I gave a talk about some of the early design decisions and mistakes I made in Claude Code. Some of the points here are already a little dated (like opening your editor!) but hopefully still entertaining. https://t.co/XT8dMsmHaY

[Adam Wolff @dmwlff]

@tonyalphaseeker Wasn't me. This is the github user dmwlff. I'm wolffiex on github: https://t.co/cmo5gdigqy

[Adam Wolff @dmwlff]

@xen_studio @Lon @trq212 You need to set CLAUDE_CODE_FORK_SUBAGENT=0 to disable it. We have paused this experiment while we investigate ways to make it more efficient.

[Adam Wolff @dmwlff]

@dani_avila7 @patricksrail Yeah! Test with CLAUDE_CODE_FORK_SUBAGENT=1 claude and let me know how it goes! Works with -p as of 2.1.121 as well.

[Adam Wolff @dmwlff]

@noseratio @LonelyWolfsh Yes, we haven't yet brought this change to Windows. That still uses ripgrep.

[Adam Wolff @dmwlff]

After 2.1.117, you may notice that Claude doesn't call its Grep or Glob Tool anymore. YES!!! It only took four months. It's faster than ever and it's all Bash.

It's so much harder to take things away than to add them. Enjoy.

[Adam Wolff @dmwlff]

Thanks for digging in here. Skill context fork isn't quite the same thing because it doesn't have perfect cache sharing. Here are some new docs: https://t.co/eA1cvW1Lu2

The new Forked Subagent experience is a couple things bundled together:
- default general agent is forked, with guaranteed cache hit
- all subagents are backgrounded
- new subagent UI below prompt input
- /fork slash command

In addition to the opt-in env var, I'm starting an experiment today. I'll let you know how it goes!

If you have feature requests, bug reports, or questions, please let me know!

[Adam Wolff @dmwlff]

@backnotprop @dexhorthy Yes, we had a crash on startup. We rolled back our pointers to 2.1.119 last night. Sorry folks! We'll be back Monday.

[Adam Wolff @dmwlff]

@59thProfile @AnthonyOdie @noahzweben this is the goal, remove the harness completely

[Adam Wolff @dmwlff]

@dexhorthy In most cases we can recognize find and grep commands as read-only.

[Adam Wolff @dmwlff]

@AnthonyOdie @eduferreyraok @noahzweben That was the hard part!! Claude's new grep is actually faster than ripgrep ;)

[Adam Wolff @dmwlff]

@AnthonyOdie @noahzweben Claude is really good at find and grep and sometimes forgets to use the custom tools. This is more natural for Claude and generally simpler is better.

Our permission system is now sophisticated enough to recognize when bash commands that Claude uses are read-only.

[ClaudeDevs @ClaudeDevs]

For the developers building with Claude, a direct line from the team.

Follow for changelogs, API releases, community updates, and deep dives.