Louis-François Bouchard

Training AI Engineers on YouTube, Substack, Towards AI Academy and O’Reilly.
Co-founder & CTO @towards_ai.
Ex-Ph.D. student @Mila_Quebec.

[Louis-François Bouchard @Whats_AI]

It's LIVE. I can finally stop saying "our internal writing benchmark".

It's called ToneBench, and it's public: https://t.co/LSZSTnRhjk

We ( @towards_AI ) were SO excited to release it first at #AIEngineerNYC, during my talk (a few minutes ago!). Thank you @swyx and the whole team @aiDotEngineer for having me!

5 months of work to answer one question: can any model write like me?

78 models + every thinking level. 6,350 scripts, each scored blind by 4 judges. That's 25,400 judgments. It's A LOT.

A few things you'll find in there right now:

- Claude Opus 5.5 is #1, and xhigh ties max for $1.00 a script instead of $2.58.
- GPT-6 Sol is the best model outside Claude, at 13 cents a script.
- GLM-5.3 Flash is 12th for 2 cents a script. Yes, 2 cents.
- Be careful with GPT as a judge. It REALLY likes itself.

You can compare any two models, see what each thinking level buys you, and read the full methodology. We share everything on how we built it, our decisions, and the full setup.

It measures my voice on my scripts, so it's one writing data point, not the final truth on which model is better. But if you write with AI, it should help you find your model.

The talk will soon be on YouTube, hopefully @swyx ? ;)

I'll keep adding every new model as it comes out, and I'll post the results here first. :)

[Louis-François Bouchard @Whats_AI]

It's specifically my tone and the typical content I produce on YouTube, which is technical and AI-related. Though I do apply a lot of storytelling aspects to make videos engaging and interesting.

But indeed, there might be some bias there, with models more used to talk about AI stuff.

[Louis-François Bouchard @Whats_AI]

Gemini 4 Carbon is the best Gemini we’ve tested for writing.

Sadly, it’s still only #27 of 87 models.

@GoogleDeepMind’s new model on ToneBench, our writing benchmark (Elo, cost per script):

Carbon low: 1590, $0.06
Carbon medium: 1774, $0.22
Carbon high: 1849, $0.59
Argon high: 1564, $0.23

Carbon Low beats Argon on high by 210 Elo for a fourth of the price.

But more thinking mostly buys length. Our tone and voice metrics stay comparable.

And high costs more than Claude Opus 5.5 on high ($0.33), which sits 600+ Elo above. DeepSeek V4.1 Flash 1017 on max beats Carbon on high for $0.014 a script.

If you’re on Google, move to Carbon on medium. Otherwise, cheaper models still write better.

https://t.co/LSZSTnRhjk

[Google Gemini @GeminiApp]

Starting today, our next frontier model Gemini 4 Carbon is rolling out to Google AI Ultra subscribers on desktop, mobile, and web at https://t.co/gTWOnOwQDE

It’s our strongest model, built to reason deeply through your most complex work:
✅ Get more thorough, reliable outputs on your toughest tasks
✅ Turn dense data and docs into polished presentations, spreadsheets, and ready-to-use reports with more precise analysis
✅ Create web apps, prototypes, and 3D scenes with stronger spatial understanding for 3D and UI design

[Louis-François Bouchard @Whats_AI]

Claude Fable 5.2 is a BIG upgrade over 5.1!

We ran @AnthropicAI’s new model on ToneBench, our writing benchmark, at all five effort settings. It’s #2 of 86 models.

On high, 5.2 is better, cheaper and faster, and it even ties 5.1 on max for about a third of the price. Even 5.2 on low beats 5.1 on high, for a quarter of the price.

Yes, thinking more seems to have a positive impact here, up to xhigh:

Low: 2416, $0.46, 1.4 min
Medium: 2426, $0.56, 1.8 min
High: 2520, $1.22, 4 min
Xhigh: 2676, $2.22, 7 min
Max: 2651, $5.75, 20 min

Xhigh is the sweet spot for us: it ties max for 39% of the price and about a third of the time.

Compared with 5.1, it seems to produce less slop, give better visual cues, be more accurate in scripts and, from high up, hit the target length. Unfortunately, it doesn’t follow our tone better than 5.1.

Surprisingly, most of the score jump comes from one of our four judges: OpenAI’s GPT-6.1 Sol. Not even from our Opus judge! Jev, our only typed judge, scores 5.2 lower than 5.1.

Max doesn’t buy anything over xhigh. A max run uses about 141K output tokens per script, 2.7x xhigh. That’s $5.75 a script, against $2.58 for Opus 5.5 on max for comparison.

If you’re on Fable, move to 5.2 on xhigh. If you’re picking a Claude model for writing, I’d still run Opus 5.5 on xhigh: it ties Fable 5.2 on xhigh for less than half the price ($1.00 vs $2.22).

Full board: https://t.co/LSZSTnRP8S

[Claude @claudeai]

Introducing Claude Fable 5.2 and Claude Mythos 5.2, designed to tackle the most daunting technical challenges.

They’re capable collaborators on ambitious projects in many fields, and show improvements in areas like coding, reasoning, and vision.

[Paul Iusztin @pauliusztin_]

@Whats_AI, CTO & Co-Founder of @towards_AI, used Claude Code and Codex to build an LLM writing benchmark. Here are 5 lessons he learned:

It's called ToneBench.

And the goal was to find out which AI models could write scripts that actually sound like him.

ToneBench includes:

• 7,050 drafts
• 8,200 judgments
• 4 LLM judges
• 9 weighted metrics

So, the lessons...

1/ Don't trust AI-generated rubrics

Your best examples should top your scoring scale.

119 of 141 model configurations scored higher than Louis's own scripts on the anti-slop metric.

The rubric penalized his actual writing habits.

He eventually rewrote most of it by hand.

2/ Measure noise before trusting sources

Both generators AND judges introduce variance.

On ToneBench, Grok 4.7 ranked:

• 5th on its best drafts
• 9th on average
• 67th on its worst

One generation can mislead you.

That's why ToneBench evaluates 5 drafts per task.

3/ Use pass/fail when possible

When binary judgments aren't enough, use a structured scale.

ToneBench judges:

• Quote evidence
• Choose 1 of 6 quality levels
• Score within that level

Each level has descriptions and examples.

4/ Don't rely on a single LLM judge

Every judge has biases.

ToneBench uses 4 judges from different families to reduce individual bias.

But you still need to measure agreement and bias.

5/ Calibrate against humans

For subjective metrics like tone and clarity, human ratings matter.

Louis is building a golden dataset of 240 examples for human evaluation, which he'll use to validate judge and rubric changes.

Calibration is still in progress.

The takeaway:

Claude Code can build your evaluation infrastructure. But it can't decide what GOOD looks like for your use case.

Start with human-rated examples, measure consistency, and validate rubric changes.

P.S. @Whats_AI' full ToneBench breakdown goes live tomorrow in Decoding AI Magazine.

Subscribe here: https://t.co/yyzLmYJZnP

[Louis-François Bouchard @Whats_AI]

@Cloudflare Full announcement here:
https://t.co/TNG6ME9Skz
https://t.co/TNG6ME9Skz

[Louis-François Bouchard @Whats_AI]

A few conversations can take you in directions you didn't expect.

A while ago, we got talking with the @Cloudflare team about AI and what they were working on. It started out of genuine curiosity about how they were approaching AI and how it could be used by businesses.

Those conversations eventually led us here.

@towards_AI is now a deployment partner for Cloudflare OS, Cloudflare's new AI workspace for companies.

We're also part of their partner announcement alongside EPAM, HappyCog, Asana, Box, Canva, GitHub and Notion.

I've been using Cloudflare myself for quite some time, and as a team, so it's pretty exciting to now be working with them.

What makes this especially interesting is the focus on security as businesses start bringing AI agents into their everyday work. It's something I believe deserves a lot more attention.

Really glad this worked out, all from wanting to hear more.

Take the lead, ask questions, be curious and interested!

[Louis-François Bouchard @Whats_AI]

I'm officially addicted to @WisprFlow, to the misfortune of my small team

amazing product team

Even in Canadian French, results are pretty 🤯

[Louis-François Bouchard @Whats_AI]

For all engineers in search of what expertise to develop as coding agents replace us, don't look for what the model cannot do; look for what the model cannot own.

There are a lot of things LLMs, agents, whatever we end up calling them, will not own for companies.

Whether it is intent, accountability, or judgment, some things require decisions and someone to answer for them.

[Louis-François Bouchard @Whats_AI]

New video on how we made LLM judges reliable!

Same model, same prompt: Grok 4.7 ranks #28 on ToneBench with its best drafts and #62 with its worst.

We measure this variability by generating the same task five times. That's the simplest of all the fixes we made.

In this one, I cover: how the whole benchmark works, and how four judges score one draft.

https://t.co/A3oUpufMzJ

[Louis-François Bouchard @Whats_AI]

Full AIE talk:
https://t.co/bNKRBIGayd

[Louis-François Bouchard @Whats_AI]

It's funny how differently people react to ToneBench.

Online, it's very hard to tell, and there’s a lot of anonymous “haters”.
In person at AI Engineer New York, people loved the talk, came to ask questions after, and a bunch connected on LinkedIn and X.

ToneBench tests whether a model can adapt to someone's tone and style and still write well.

On X, the usual comment is something like "your LLM judges are biased, Gemini is better."
But when people hear the full story of how it works, they get it.
For example, every model writes the same draft 5 times, so we can see how much it changes from one generation to the next.

And we worked hard on making it the least biased possible, even as it runs with LLM judges (4 specifically!).

A lot of people asked me why it matters to them if it's my voice being tested.

If a model can adapt to my style, there's a good chance it can adapt to yours.

So use it to find the top 2 models, then try them on your own writing and pick the one you like.

The full talk is live on the AI Engineer YouTube channel, including what it cost us to build (one prompt change now costs over $4,000 😅).

[Louis-François Bouchard @Whats_AI]

As AI engineers, we spend weeks picking the right model, tuning prompts and building evals.

It works locally when we rely on powerful models, a stable connection and our controlled environment.

But as soon as we do not have full control over each variable, our agent dies 6-7…20 hours into a task because a server returns a 504 timeout error.

Good products like Codex, Claude Code or Cursor manage to not have this happen. Of course, else their products would be unusable for longer tasks.

So, we know it's possible to mitigate the problems that come with more and more complex long running tasks; both in terms of context management and workload management. But how can you do that on your own, in a small company or even just as your average AI builder?

Fortunately, some companies specialize in this challenge, which is the case with @inngest, literally described as building “durable execution infrastructure for background jobs, complex event-driven workflows, and AI agents.” So I was super excited to have the opportunity to interview their CEO @itstonyhb :)

Read the full interview 👇
https://t.co/M7fMbMobtm

[Louis-François Bouchard @Whats_AI]

@jurlycat Thank you !!

[Louis-François Bouchard @Whats_AI]

No matter how many talks I give, I still feel anxious right before going on stage.

A year ago, that feeling was enough to make me say no. To talks, to events, even to running a race with thousands of people “watching”.

But what then often happens is I remember the worst part is always the moment before. If I get through it, the rest is fine, and the best part comes right after.

So this year I kept saying yes.

Around 10 conferences across London, Switzerland, San Francisco and New York, plus a bunch of online events and podcasts. My first half marathon in Montreal, finished in 1:45:33. And writing our second book on AI engineering.

If you’re someone who keeps saying no because of that feeling, please keep showing up.
You’ll likely end up surprising yourself.

[Louis-François Bouchard @Whats_AI]

@cwolferesearch Indeed! Quite a painful process (more to some than others), but highly rewarding!

Proud of you for going through that too my friend!!

[Louis-François Bouchard @Whats_AI]

@ThePracticalDev Full recording:
https://t.co/ohZYcvYpaU

[Louis-François Bouchard @Whats_AI]

Which open model writes the best?
That’s what I answered at Hacktoberfest’s live event on @ThePracticalDev Community this week.

Thousands of people joined, and it ended up being the most-viewed guest session of Hacktoberfest on DEV so far.

I built ToneBench because AI writing still sounds like AI, even from the best models.
It tests open and closed models on the same writing tasks, so you can see which one sounds the most human before you pick one.

The recording is now up on MLH’s Twitch. If you missed it, I added the full recording link in the comments.

Thanks to DEV Community and @MLHacks for having me.

[Louis-François Bouchard @Whats_AI]

Join us live here: https://t.co/dzaKi92Hy8

[Louis-François Bouchard @Whats_AI]

I'm joining @Al_Grigor on the @DataTalksClub podcast this Monday, November 2 at 9am EST.

We'll talk about how to ship reliable LLMs and the rise of AI Engineering.

We’re also going to discuss:

- What it takes to become an AI engineer today
- Shipping LLMs that hold up in production
- Where vibe coding and agents actually fit

If you're building with LLMs, this one's for you!

To join live, I've added the registration link in the first comment.

[Thariq @trq212]

very weird thing happening rn

technical people are sad, they think their skills are now useless

non-technical people are desperately wishing they were more technical so they could use agents better