Alexis Galbourdin

Freelance Technical Consultant | 15+ years of experience | Strategy, Architecture, Delivery

[Alexis Galbourdin @agalbourdin]

My HighWalk Benchmark is out!
I updated it last minute to add Opus 5, which changed the rankings quite a bit as it achieved top quality in its high effort.

This benchmark is focused on updating technical specifications from 46 commits on a Laravel project.
It requires strong code analysis, technical abstraction and writing skills, which in my opinion set it apart from coding or more general benchmarks.
It confirms my previous observations that some strong performers in code were not always the best suited for this, especially in the open-weight field.

Some key takeways:

- A higher reasoning effort does not guarantee a better result: Opus 5 performs better in high than in xhigh, a trend also observed with Sonnet 5, DeepSeek V4 Flash/Pro, and GPT 5.6 Luna/Terra. Opus 4.8 shows the opposite.

- Grok 4.5 (high) emerges as the best overall thanks to an unmatched combination of quality and operational efficiency.

- Opus 5 (high) is the best in quality by a significant 10% with zero hard failures, making it a very solid choice. It outperforms Fable 5 while costing less.

- GLM 5.2 is the best open-weight model in the rankings, both overall and in quality, outperforming Kimi K3 which suffers from weaker operational efficiency as well as a lower quality.

- GPT 5.6 Sol (xhigh) achieves one of the best raw-quality scores, but its operational efficiency dropped to zero after exceeding the budget and time caps.

- Some mid-range models, Claude Sonnet 5 and GPT 5.6 Terra, perform very well.

Of course this benchmark covers one use case, one project and one reference document. I did not test all possible combinations of models and effort levels, and it's not meant to be a guarantee in other contexts.

It helps me get a clearer picture of model performance for my use cases, and hopefully it will be relevant to others as well!

[Alexis Galbourdin @agalbourdin]

Been looking for a free STT setup on Mac that handles French + English technical terms well.

Local Parakeet and Whisper Large weren't good enough on their own. Grok STT 1.0 was pretty bad, while GPT-4o Transcribe was much better - but requires an API key.

Tried Handy, OpenWhisper, Spokenly, and TypeWhisper. Apple Intelligence post-processing in Spokenly/TypeWhisper wasn't great either, and a local LLM would use too much RAM just for this.

Then I discovered TypeWhisper supports OpenAI post-processing through a ChatGPT subscription.

Local Parakeet + Luna Low + a custom cleanup prompt now gives me very good results with acceptable latency. And since Luna Low barely touches my quota, it's essentially free with my existing subscription.

Is there another setup I'm missing that could work even better?

[Alexis Galbourdin @agalbourdin]

It seems that Qwen 3.8 Max is getting mixed reviews on coding benchmarks.

Funny enough, the models that usually get hyped (like Kimi K3) disappointed me on technical spec updates… while Qwen 3.8 Max did extremely well on HighWalk.

This is precisely why I created it.

Full results coming soon.

[Alexis Galbourdin @agalbourdin]

@allanmacgregor Checked an agency’s site I’m working with: founder vibecoded it.
Content API wide open (write access for anyone), client-side i18n, broken cookie consent… the list goes on. Lots of work ahead in the "audit & fix vibecoded apps" space!

[Alexis Galbourdin @agalbourdin]

If I remember correctly, about a year ago, Perplexity argued in its debate with Cloudflare that an AI agent acting on behalf of a user shouldn't be treated like a traditional web crawler.

Now, a US appeals court has recognized a similar distinction: when an agent acts on behalf of a user, the access may legally be attributed to the user.

The flip side is just as interesting: if an agent is legally treated as acting on behalf of its user, what happens when it acts unexpectedly or crosses a line? That's not what this ruling decides, but it's still an important open question.

[TNW @thenextweb]

Amazon said the agent broke in. The court called it the user. The reasoning matters more than the ruling, because if an agent acting on your instruction is legally you, every platform just lost its cleanest way to block them. https://t.co/6QOGpMqilG

[Alexis Galbourdin @agalbourdin]

Working on an Agent Skill that needs a few project-specific files including:
- Variables (no secrets)
- State (to track the last execution)
- Learnings (to keep track of problems encountered and their solutions)

These files must live outside the Skill directory so the Skill itself stays stateless and can be safely updated with npx skills.

First question is their location. I’m currently leaning toward something like .config/skills/{skill}/ at the repo root. I haven’t found any established convention for this.

Also wondering about the initialization method:

- On first invocation the agent notices the files are missing, asks the user for the required info, then creates them.
=> Best UX, zero friction. But it means using the LLM for a purely deterministic setup task, and it complicates the Skill itself.

- Provide an interactive shell script that the user runs once before using the Skill.
=> Cleaner separation of concerns and keeps the agent out of deterministic work. But it adds a manual step.

Curious what others prefer in this kind of setup.

[Alexis Galbourdin @agalbourdin]

I tested the updated DeepSeek V4 Flash (0731) against HighWalk, interesting results:
- High effort gave the best quality
- It did way better than the previous version (both in high effort), costing the same
- Same raw quality score as Kimi K3 max but costing far less
- Still behind GLM 5.2 in quality

Full results: https://t.co/BZbzOr4izz

[Alexis Galbourdin @agalbourdin]

I couldn't find a screenshot beautifier that was both simple and flexible.

Most existing tools are packed with options, require an account or don't let you use your own background image.

So I built FrameUp: https://t.co/XGr10ODIEq

- Built-in backgrounds or upload your own
- 100% client-side
- Free & no account required

[Alexis Galbourdin @agalbourdin]

Merci d’avoir relayé le benchmark !

« les commits Laravel sont publiquement accessibles et auraient pu figurer dans les données d’entraînement »
=> Pour préciser ce point, il s’agit de commits d’un projet spécifique dont le code n’est pas public. C’est un projet Laravel mais les critères sont des changements très spécifiques au projet et à ses règles métiers.

https://t.co/yuosB3kA1b

[Alexis Galbourdin @agalbourdin]

My HighWalk Benchmark is out!
I updated it last minute to add Opus 5, which changed the rankings quite a bit as it achieved top quality in its high effort.

This benchmark is focused on updating technical specifications from 46 commits on a Laravel project.
It requires strong code analysis, technical abstraction and writing skills, which in my opinion set it apart from coding or more general benchmarks.
It confirms my previous observations that some strong performers in code were not always the best suited for this, especially in the open-weight field.

Some key takeways:

- A higher reasoning effort does not guarantee a better result: Opus 5 performs better in high than in xhigh, a trend also observed with Sonnet 5, DeepSeek V4 Flash/Pro, and GPT 5.6 Luna/Terra. Opus 4.8 shows the opposite.

- Grok 4.5 (high) emerges as the best overall thanks to an unmatched combination of quality and operational efficiency.

- Opus 5 (high) is the best in quality by a significant 10% with zero hard failures, making it a very solid choice. It outperforms Fable 5 while costing less.

- GLM 5.2 is the best open-weight model in the rankings, both overall and in quality, outperforming Kimi K3 which suffers from weaker operational efficiency as well as a lower quality.

- GPT 5.6 Sol (xhigh) achieves one of the best raw-quality scores, but its operational efficiency dropped to zero after exceeding the budget and time caps.

- Some mid-range models, Claude Sonnet 5 and GPT 5.6 Terra, perform very well.

Of course this benchmark covers one use case, one project and one reference document. I did not test all possible combinations of models and effort levels, and it's not meant to be a guarantee in other contexts.

It helps me get a clearer picture of model performance for my use cases, and hopefully it will be relevant to others as well!

[Alexis Galbourdin @agalbourdin]

@XFreeze Thanks! Full breakdown and link to the benchmark details for anyone interested:

[Alexis Galbourdin @agalbourdin]

My HighWalk Benchmark is out!
I updated it last minute to add Opus 5, which changed the rankings quite a bit as it achieved top quality in its high effort.

This benchmark is focused on updating technical specifications from 46 commits on a Laravel project.
It requires strong code analysis, technical abstraction and writing skills, which in my opinion set it apart from coding or more general benchmarks.
It confirms my previous observations that some strong performers in code were not always the best suited for this, especially in the open-weight field.

Some key takeways:

- A higher reasoning effort does not guarantee a better result: Opus 5 performs better in high than in xhigh, a trend also observed with Sonnet 5, DeepSeek V4 Flash/Pro, and GPT 5.6 Luna/Terra. Opus 4.8 shows the opposite.

- Grok 4.5 (high) emerges as the best overall thanks to an unmatched combination of quality and operational efficiency.

- Opus 5 (high) is the best in quality by a significant 10% with zero hard failures, making it a very solid choice. It outperforms Fable 5 while costing less.

- GLM 5.2 is the best open-weight model in the rankings, both overall and in quality, outperforming Kimi K3 which suffers from weaker operational efficiency as well as a lower quality.

- GPT 5.6 Sol (xhigh) achieves one of the best raw-quality scores, but its operational efficiency dropped to zero after exceeding the budget and time caps.

- Some mid-range models, Claude Sonnet 5 and GPT 5.6 Terra, perform very well.

Of course this benchmark covers one use case, one project and one reference document. I did not test all possible combinations of models and effort levels, and it's not meant to be a guarantee in other contexts.

It helps me get a clearer picture of model performance for my use cases, and hopefully it will be relevant to others as well!

[Alexis Galbourdin @agalbourdin]

@0xAndre__ @cb_doge @elonmusk Agree, this is a very practical task that I run weekly on multiple projects.
I built HighWalk after noticing how differently models performed on this compared to coding benchmarks.
Useful for tracking progress going forward!

[Alexis Galbourdin @agalbourdin]

@Dawn23871512983 @cb_doge One pass, that’s why operational efficiency shouldn’t be looked at in isolation, it needs to be combined with raw quality.
The global ranking is an 80/20 mix of raw quality and operational efficiency.

[Alexis Galbourdin @agalbourdin]

@vishnuv_786 @teslaownersSV Correct, the ranking shown in this image is wrong.
3rd is GPT-5.6 Terra, 4th Sonnet 5, 5th GLM 5.2.
Original (correct) chart here:

[Alexis Galbourdin @agalbourdin]

My HighWalk Benchmark is out!
I updated it last minute to add Opus 5, which changed the rankings quite a bit as it achieved top quality in its high effort.

This benchmark is focused on updating technical specifications from 46 commits on a Laravel project.
It requires strong code analysis, technical abstraction and writing skills, which in my opinion set it apart from coding or more general benchmarks.
It confirms my previous observations that some strong performers in code were not always the best suited for this, especially in the open-weight field.

Some key takeways:

- A higher reasoning effort does not guarantee a better result: Opus 5 performs better in high than in xhigh, a trend also observed with Sonnet 5, DeepSeek V4 Flash/Pro, and GPT 5.6 Luna/Terra. Opus 4.8 shows the opposite.

- Grok 4.5 (high) emerges as the best overall thanks to an unmatched combination of quality and operational efficiency.

- Opus 5 (high) is the best in quality by a significant 10% with zero hard failures, making it a very solid choice. It outperforms Fable 5 while costing less.

- GLM 5.2 is the best open-weight model in the rankings, both overall and in quality, outperforming Kimi K3 which suffers from weaker operational efficiency as well as a lower quality.

- GPT 5.6 Sol (xhigh) achieves one of the best raw-quality scores, but its operational efficiency dropped to zero after exceeding the budget and time caps.

- Some mid-range models, Claude Sonnet 5 and GPT 5.6 Terra, perform very well.

Of course this benchmark covers one use case, one project and one reference document. I did not test all possible combinations of models and effort levels, and it's not meant to be a guarantee in other contexts.

It helps me get a clearer picture of model performance for my use cases, and hopefully it will be relevant to others as well!

[Alexis Galbourdin @agalbourdin]

@cozybearlog @cb_doge Interestingly, GPT-5.6 Sol was the only model that actually spotted and mentioned a contradictory code behavior in the chat, even though it wasn’t asked to.
I should probably highlight this in the benchmark results!

[Alexis Galbourdin @agalbourdin]

@cozybearlog @cb_doge Thanks!
The agent is only tasked with updating the main spec file, it’s not responsible for code comments.
It does get penalized if it modifies the wrong section or introduces something irrelevant or incorrect.

[Alexis Galbourdin @agalbourdin]

Exactly! I built HighWalk because I kept seeing a big gap between how models performed on this task in real day-to-day use versus the results (and sometimes the hype) around new releases.

I wanted a reliable way to test this and track how future models improve.

Hopefully open-weight models will get better at it.

[Alexis Galbourdin @agalbourdin]

Nice to see this circulating.

I’ll maintain the benchmark and update it with new models regularly.

For full chart + details, here’s the original:

https://t.co/YmBEpqwlhh

[DogeDesigner @cb_doge]

BREAKING: Grok 4.5 (high) ranks #1 on the HighWalk benchmark, which tests how well AI agents update technical specifications from code changes.

Grok delivered the best combination of quality and operational efficiency, finishing ahead of Claude and GPT.

[Alexis Galbourdin @agalbourdin]

@teslaownersSV Thanks for the highlight!
Full chart, more context and benchmark link here:

https://t.co/YmBEpqwlhh

[Alexis Galbourdin @agalbourdin]

My HighWalk Benchmark is out!
I updated it last minute to add Opus 5, which changed the rankings quite a bit as it achieved top quality in its high effort.

This benchmark is focused on updating technical specifications from 46 commits on a Laravel project.
It requires strong code analysis, technical abstraction and writing skills, which in my opinion set it apart from coding or more general benchmarks.
It confirms my previous observations that some strong performers in code were not always the best suited for this, especially in the open-weight field.

Some key takeways:

- A higher reasoning effort does not guarantee a better result: Opus 5 performs better in high than in xhigh, a trend also observed with Sonnet 5, DeepSeek V4 Flash/Pro, and GPT 5.6 Luna/Terra. Opus 4.8 shows the opposite.

- Grok 4.5 (high) emerges as the best overall thanks to an unmatched combination of quality and operational efficiency.

- Opus 5 (high) is the best in quality by a significant 10% with zero hard failures, making it a very solid choice. It outperforms Fable 5 while costing less.

- GLM 5.2 is the best open-weight model in the rankings, both overall and in quality, outperforming Kimi K3 which suffers from weaker operational efficiency as well as a lower quality.

- GPT 5.6 Sol (xhigh) achieves one of the best raw-quality scores, but its operational efficiency dropped to zero after exceeding the budget and time caps.

- Some mid-range models, Claude Sonnet 5 and GPT 5.6 Terra, perform very well.

Of course this benchmark covers one use case, one project and one reference document. I did not test all possible combinations of models and effort levels, and it's not meant to be a guarantee in other contexts.

It helps me get a clearer picture of model performance for my use cases, and hopefully it will be relevant to others as well!