Shane Butler

😴☕️☕️💻🥃🥃😴

[Shane Butler @ShaneAusley]

There's a window right now where very few data professionals understand this.

[Shane Butler @ShaneAusley]

The best thing you can do for your career as a data professional right now is unrelentingly attempt to automate yourself out of your current role.

[Shane Butler @ShaneAusley]

The fastest way to make yourself irreplaceable is to try as hard as you can to make yourself replaceable. , because the person who automates their own role doesn't get replaced. They become the most valuable person in the room.

[Shane Butler @ShaneAusley]

For the companies now basing performance review on "ai usage", why not just ask for your employees for their chat gpt wrapped ;)

[Shane Butler @ShaneAusley]

2026 bet: teams scaling AI products will pour money and resources into quality evals, but still won't be able to show revenue or retention lift as result of AI efforts. Huge inference gap.

[Shane Butler @ShaneAusley]

The most high-performing data teams are those that are relentlessly unwilling to accept a broken way of working.

[Shane Butler @ShaneAusley]

Teams spend a lot of time aligning on metrics, but the real alignment should happen one step earlier: on the actual questions the metrics are supposed to answer.

[Shane Butler @ShaneAusley]

Actual data-driven teams don’t align on metrics. They align on the questions the metrics are meant to answer.

[Shane Butler @ShaneAusley]

AI doesn’t fail because it’s "wrong".
It fails because teams fail to measure it.

[Shane Butler @ShaneAusley]

The more general ai systems become, the more their behavior depends on context, nuance, and task-specific expectations. Even simple-looking outputs can take experts minutes (or hours) to judge, and most evals fail to capture the full complexity of how these models behave in prod.

[Shane Butler @ShaneAusley]

We’re quickly reaching a point where evaluating the work is harder than producing it.

[Shane Butler @ShaneAusley]

AI labs estimating AI’s impact on productivity need to ground their measurements in real human data, not in guesses generated by the model under evaluation. @AnthropicAI's approach doesn’t measure human behavior. It measures its model’s internal assumptions about human behavior.

[Anthropic @AnthropicAI]

New Anthropic research: Estimating AI productivity gains from Claude conversations.

The Anthropic Economic Index tells us where Claude is used, and for which tasks. But it doesn’t tell us how useful Claude is. How much time does it save?

[Shane Butler @ShaneAusley]

I’ve been spending more time working in public as I distill what a practical AI evaluation lifecycle actually looks like. Rough thoughts:

https://t.co/U3Ce6kjSfR

[Shane Butler @ShaneAusley]

GenAI is the fastest-adopted technology in history, and that speed is exactly why we’re behind on AI evaluation. Adoption outran the skill base. Teams shipped before they had methods to measure. If AI is going to be the foundation of software, evaluation has to catch up fast.

[Shane Butler @ShaneAusley]

@TERSUD_SNCF @TERSUD_SNCF we’ve been stuck on the tracks on TER 86054 for 1hr20min minutes with locked windows, locked WC, and no AC. Please let us out ...

[Shane Butler @ShaneAusley]

@TERSUD_SNCF we’ve been stuck on the tracks on TER 86054 for 1hr20min minutes with locked windows, locked WC, and no AC. Please let us out ... and bring wine

[Shane Butler @ShaneAusley]

@principal is your site down? I'm getting 'Error 503 Backend unavailable, connection timeout'

[Shane Butler @ShaneAusley]

@Fly_Airlink Hey I’m not getting a response from [email protected]. Is there an email direct to a specific person I can contact regarding booking with my flight credits? I tried calling as well, but just sat on hold for 30 min with no answer.