Adam Wolff

Claude Code @AnthropicAI 🤖
Avid cook, dedicated snow person, yoga enthusiast

[Adam Wolff @dmwlff]

I ended my time at @Meta as a director.

But I started as an engineer on FB Chat.

Everything about it was broken — we had to rewrite it.

And while the effort to fix it is one the projects that led to @reactjs, the most important fix was far simpler...

Here’s the full story:

—

I worked on Facebook Chat for several years, both on the front end and the infrastructure.

Before the major effort to redo the UI, FB Chat was super broken and we had no idea why.

We got tons of bug reports about Chat being broken every day, but we noticed an odd pattern in the data: the volume of reports didn’t match the volume of usage. It was time-shifted from the peaks we’d see in the US.

We didn’t know what was wrong, but we knew the code was a mess.

We set about rewriting both the front-end and the back-end in an effort to fix it.

The front-end rewrite pulled in a whole team of amazing engineers and became one of the big threads that led to @reactjs

In the public eye, we portrayed this project as the one that ultimately fixed Chat.

And the way I’ve usually told it, fixing Facebook Chat and the birth of React are the same story.

But no framework was going to fix the worst problem with Chat.

—

During the time we were working on the Chat rewrite, we were also replacing the original Erlang backend with one written in C++.

This was probably a good move, but the problem wasn’t with Erlang either.

Our initial spec for the new backend didn’t say much about observability, but it was an important feature, and the rewrite forced us to rebuild it.

Little did we know this would lead us to the root cause of our problems…

When we finally gained insight into our deliverability data, we were able to cut it by region.

We noticed Chat was really popular in India. This was before WhatsApp, at a time when SMS wasn’t reliable.

Eventually we pinpointed a region in India where one specific DNS provider was giving out the wrong IP addresses for our Chat servers.

So when people went to use Chat, they would sometimes get a notification that they had a message, and then it would disappear.

Or they’d send a message and it would get lost. All because they were connecting to the wrong IP address.

That was it!

None of the sexy new tech we were working on was going to solve that problem.

Ever.

—

Instead, the solution was to build observability that allowed us to track end-to-end message delivery.

In the end, we could start with a broad cut of our data by country or web browser, and then zoom all the way in to look at what happened to a specific message for a specific user.

Once we pinpointed that the problem was with a DNS server, the matter was resolved with a quick phone call. I don’t know what they did, but I imagine it was something like turning it off and turning it on again.

We sometimes talk about observability as if it’s enough to buy a product like Datadog and just look at the pretty graphs.

Sure, that’s a start.

But true observability is a feature that needs to be built— painstakingly, iteratively, by-definition starting with a shot in the dark.

—

These days, it has become fashionable to poo-poo the idea of being data-driven.

People point out that measurement can distort the phenomenon that is being observed.

They want to make processes “data-informed.”

But this seems like silly backlash against the only rigorous standard in all of software engineering:

That we hold ourselves to an objective standard.

We measure how long things take, how many errors we encounter, how often a process successfully runs to completion.

So here’s what this experience taught me about observability:

When an issue happens in production, time-box the investigation.

Sure, take a few hours to try and figure it out by looking in the logs and inspecting the code.

But if you’re coming to the end of the day and you still don’t have a fix, then push a PR that adds logging.

The first one may be just a guess, but it will begin a process that leads to the truth.

And that is what we should all ultimately be striving for.
—
For more engineering tips and stories, follow me @dmwlff

[Adam Wolff @dmwlff]

@BuilderOfAgents mid-turn should be handled, but there may be bugs. let me investigate.

[Adam Wolff @dmwlff]

@_catwu Great working with you @_catwu. I'll miss you.

[Adam Wolff @dmwlff]

@steffensiewert This is surprisingly difficult. By definition, the effort selector is meant to represent exactly the part of task value that isn't knowable from the prompt.

[Adam Wolff @dmwlff]

ultrathink is back in the CC CLI.

We turned this off for a while because it's weird and maybe confusing.

But I like weird, and I don't think it's that confusing.

I mostly run Opus 5.5 on medium. Some turns, though, I want to put a little extra mustard on it. ultrathink!

[Adam Wolff @dmwlff]

@patrickc Now make it so you can point to something and give feedback about that.

[Adam Wolff @dmwlff]

@MohamedHatem728 It's always xhigh, so nothing if you already there and it'll actually lower it if you are on max (I think, haven't tested this.)

[Adam Wolff @dmwlff]

@Bayesian0_0 exactly!!

[Adam Wolff @dmwlff]

@swangentr ofc!

[Adam Wolff @dmwlff]

Yeah, fair. This is a little glib. Unfortunately the whole story is too complex to tell in a catchy tweet.

I work on model performance for Claude Code. I study the effects of effort and different models on accuracy and cost. I was part of the team that pushed hard to make "medium" the default for Opus 5.5.

In general, increasing effort may not do what you think it does. Sometimes it helps Claude be more thorough or catch edge cases, but often it leads to overthinking or scope expansion that can be unhelpful.

I generally think you are better off taking more turns at medium than running at xhigh all the time. But there are still times I personally miss xhigh, especially at the beginning of session where I'm researching the state of a feature or bug before settling on the fix.

Given that our new models support changing effort without breaking cache, I thought it would be fun to bring this feature back. But it *is* confusing, and there are good reasons we turned off, mostly because it is hard to explain.

Sometimes it takes a few tries before even a good idea for a feature sticks. I thought it was time to give this one another try. I'm hoping that people will play with it and let me know what they think. Even though I know the stakes are high when it comes to cost and rate limits, we also gotta keep having fun.

[Adam Wolff @dmwlff]

@dkodr It's quite confusing and you're right that the rainbow coloring has persisted for a lot of people. Suffice it to say that before recent versions, it didn't quite work right. Newer models support per-turn-effort better and that's what we fixed here.

[Adam Wolff @dmwlff]

@proxy_vector yup! whatever it was set to before that turn.

[Adam Wolff @dmwlff]

@UsoToChinmoku that's the nice thing about it, it's temporary for just that turn

[Adam Wolff @dmwlff]

@someRandomDev5 @mdp_sec no! newer anthropic models have dynamic effort!

[Adam Wolff @dmwlff]

@UsoToChinmoku Oh yup, it's same thing, just more fun

[Adam Wolff @dmwlff]

@Keshavatearth sorry, I didn't mean to obtuse. it raises effort to xhigh for one turn.

[Adam Wolff @dmwlff]

@mdp_sec it's just an effort change to xhigh for one turn. any subagents spawned in that turn will also run at xhigh.

[Adam Wolff @dmwlff]

@Foocux coming soon

[Adam Wolff @dmwlff]

@lacker @badlogicgames see, this is why you don't want to yell at the agents! you'll habituate yourself to yelling.

[Adam Wolff @dmwlff]

@thdxr I wonder if this is the end-state for both compaction and memory.

[Adam Wolff @dmwlff]

@lacker @badlogicgames yeah it's great to give blunt feedback, but expressing frustration and negative emotion hurts model performance and debases the user