Turns out DeepSeek harness improves the score of DeepSeek V4.1 Flash by 2-3 points on average.
It also costs 80% more due to subagents and more output tokens.
New results:
DeepSeek V4.1 Flash 1017 + Claude Code harness:
- max, mean of 7: 39.1
- high, mean of 5: 36.4
- low, mean of 5: 34.4
DeepSeek V4.1 Flash 1017 + DeepSeek harness:
- max, mean of 7: 42.7
- high, mean of 5: 39.2
- low, mean of 5: 34.8
For comparison:
GPT-6 Astra, max: 45
Opus 5.5, max: 41.7
Muse Wonder Full 1.0, max: 42
It's a crazy model, and the only one catched cheating. All the results are available on the benchmark website.
This is wrong. China to this day doesn’t have a Mythos-equivalent model. Mythos was trained in February. They are at a minimum 9 months behind and have been consistently since the release of DeepSeek in Jan 2025