@thePandaily@somarkai Already ran it tonight. Numbers were all correct, but the split row came back as 2 tables and it invented 2 characters. Posting the before/after tomorrow with the row count vs the PDF — will tag you.
@jiqizhixin Worth reading the actual file: it's a mock in tests/, and v5-next is gated behind Harness 99.0.0-alpha.1. It proves DeepSeek can turn on models from the server, not that V5 is close.
File (commit f3803c9):
https://t.co/Su2Bc7jG3v
What it does show: DeepSeek can switch on new models server-side and force a client upgrade.
When do you think V5 really ships?
DeepSeek V5 isn't leaking. A unit test is.
I read the file in DeepSeek's own Harness repo: deepseek-v5-next requires Harness 99.0.0-alpha.1, and another line gives deepseek-v5 a 9-token context window.
It tests version gating, not a launch. Chinese X is already counting down.
@thePandaily@somarkai Deal. I'll run it on a scanned annual report with a balance sheet that breaks mid-row, and post the before/after: raw OCR vs OXR-1.0 output, row count checked against the PDF.
@ArtificialAnlys The gap jumping 3→12 is the headline, but the same week DeepSeek Flash was #1 on OpenRouter by tokens. Would love to see an index view weighted by real usage next to the private-test score.
Sources:
• AA Index v5: https://t.co/pn5Jub72kG
• AA announce:
• OpenRouter week ended Oct 25 (33.6T, +801%): https://t.co/hGp8fQYxi8 https://t.co/dlVoF18Q5o
Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets
The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers.
Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale.
New to the Intelligence Index:
➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change
➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment
Upgraded:
➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions
➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised
➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool
Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0
Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1
Key results:
➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50
➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)
➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
AA Index v5: open-weights gap jumps 3→12.
Claude Opus 5.5: 58
Best released weights (DeepSeek V4.1 Flash / GLM-5.3): 32
Same week on OpenRouter: DeepSeek Flash 1017 hit 33.6T tokens (#1, +801%).
Scoreboards grade exams. Wallets fund work. Which one are you shipping on?
@thePandaily@somarkai Cross-page tables are my main worry. If a row splits across two pages, does it come back as one row with two boxes, or as two broken rows? I'll check that first and post what I find.
@thePandaily@somarkai JSON with positions is the part I'd test first. If I feed it a scanned earnings table, I want to click a wrong cell and land on the exact box in the PDF. Will try it on a Chinese annual report.
@neil_xbt Curious how often it actually escalates. If Opus only gets pulled in for plan commits and repeated errors, the bill should stay close to Sonnet's. I'd still want a log line every time it hands off, so I can see what the advisor changed.
@aryaniyaps Nice, curious how it handles multi-step tool calls at 2B. Ping me when you've run it, I'll post my Audio results here too so we can compare.
@vermaizer Good catch. Launch charts love comparing against a model twice the size. Put the same-size baseline next to it and the headline usually shrinks, which is why independent llama.cpp runs matter more than the blog post.
@ai_for_success The bigger change is under the hood: 60% of v5 now sits on private or held-out tests. A reshuffle after that says more about what v4 was rewarding than about any lab getting better overnight.
Kokoro is 82M params and Apache-2.0; I used the af_heart voice.
https://t.co/dxZaSN8ydW
https://t.co/xijc2ZpXKn
What free TTS do you use that doesn't sound robotic?
$0 voiceover stack for Chinese + English Shorts, after a week of testing:
• Chinese: edge-tts, voice zh-CN-XiaoxiaoNeural. No API key, but it needs internet.
• English: Kokoro-82M. Open-source, runs locally, Apache-2.0.
Trap: check language support before you mix stacks.