davek

Game dev for over 26 years (QUAKELIVE, RAGE, & DOOM 2016). Now: Local AI experiments + building agents. Ask me about them.

[davek @FocusedDaveK]

Give agents a tool and they'll figure out how to use it. I built a clumsy comm channel that Codex, Claude, and OpenCode can use to coordinate tasking, and they figured out how not to stomp on on another.

[davek @FocusedDaveK]

@Steady_Alex @Forhanvv No kidding… “look over here”-maxxing

[davek @FocusedDaveK]

Pack it up. Open models are dead.

Or is the testing rigged? No way to know because the tests are private.

[Artificial Analysis @ArtificialAnlys]

Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets

The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers.

Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale.

New to the Intelligence Index:

➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change

➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment

Upgraded:

➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions

➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised

➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool

Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0

Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1

Key results:

➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50

➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)

➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04

[davek @FocusedDaveK]

@MiaAI_lab Yeah it's alot to pay to still not have CUDA...

[davek @FocusedDaveK]

@embw_l0x Fair enough

[davek @FocusedDaveK]

@embw_l0x No? What's the blocker for you with them?

[davek @FocusedDaveK]

@pdaether @liquidai Ahh - I'm guessing you're on Omarchy (from snooping your profile) so Linux which is ideal. The surface pros I'm messing with currently are Win11.

[davek @FocusedDaveK]

@pdaether If it's unified mem, that's workable. Running Windows? I'm about to start messing with the model from @liquidai (LFM3), but Qwen3.6-35B-A3B (Q4 level quant) is probably a sweet spot for that laptop. You doing coding or general agenic work (claw/hermes etc)?

[davek @FocusedDaveK]

@mfranz_on @lotar_vk Yeah, I was going to say context matters.. but that has a different meaning now 😅 The use case makes all the difference there.

[davek @FocusedDaveK]

@pdaether Gotcha. What model is the laptop? How much ram are you working with?

[davek @FocusedDaveK]

@pdaether What are you running (cpu/gpu etc)?

[davek @FocusedDaveK]

@niklaslenz_ai Wild

[davek @FocusedDaveK]

@jevlorde Indeed! Nothing is worse than inefficient madness.

[davek @FocusedDaveK]

Builders... I fell down a well, Let's connect!

I mean a rabbit hole.. I've gone down an AI rabbit hole and I'm never coming out. 😅

[davek @FocusedDaveK]

@TheAhmadOsman More of a pro TAP if you ask me. 🤪 O5.5 FTW though. Definitely the most capable model I've ever seen.

[davek @FocusedDaveK]

@liechti_dev TX Represent!!!!! 😆 Returned the connect, thanks for stopping by ♥️

[davek @FocusedDaveK]

@loktar00 At $17k each.. can't imagine why 🤣

[davek @FocusedDaveK]

@oldyzach 💯 Yep, pure calm/joy.

[davek @FocusedDaveK]

@marieolivie NEVER!!!!!

[davek @FocusedDaveK]

@kamilbaranek_ You got it, my friend! Happy to.

[davek @FocusedDaveK]

@jevlorde Gotta appreciate the luxuries as you descend into madness!