Ghost In The Stack

Local AI · Stretching a Single RTX 4090 24GB
Form Over Function

[Ghost In The Stack @GhostInTh3Stack]

Okay, they've fixed it
https://t.co/CvyP7bICZ5

[Artificial Analysis @ArtificialAnlys]

Announcing Artificial Analysis Intelligence Index v4.2. We are accelerating elements of our upcoming v5 release with interim updates to keep pace with the frontier. Index v4.2 has more complex and realistic tasks, and more private test sets to prevent gaming

Intelligence Index v4.2 changelog:

+ AA-Briefcase, our agentic knowledge work evaluation with a private test set

+ @HelloSurgeAI's GDP.pdf, long context document reasoning across 4,592 PDF pages

- GPQA Diamond, an exceptional scientific reasoning evaluation that has now been saturated

… plus greater weighting on held-out test sets to prevent gaming, and grading infrastructure upgrades to increase robustness

This update brings the Index closer to real-world use cases with more challenging, complex and realistic tasks and private test sets to prevent gaming. We have been planning and building elements of Index v5 for months - it’s been 8 months since we launched Index v4 in January.

We have deliberately held back updates to keep the Index stable through recent major model launches. However, with the frontier moving so quickly in the past weeks, we feel it is important to deliver an immediate interim update to ensure our Index remains as relevant and useful as ever to users.

Beyond this interim update, our team is hard at work on v5 of the Index. We are planning more incremental releases in the near future. Stay tuned!

Intelligence Index v4.2 changes in detail:

➤ Adding AA-Briefcase: Our in-house evaluation with a private held-out test set, AA-Briefcase tests models on realistic agentic knowledge work tasks in complex projects built by industry experts. Models are evaluated on multi-week knowledge work projects, each with many linked tasks and thousands of input source files. AA-Briefcase combines rubric and pairwise grading to evaluate verifiable task success, analytical quality, and presentation quality, giving a holistic view of overall agentic capability in knowledge work.

➤ Adding GDP.pdf: Created by @HelloSurgeAI, GDP.pdf evaluates single-turn professional document reasoning across 100 PDFs and ten domains. Models must synthesize evidence distributed across 4,592 pages, including text, tables, charts, footnotes, and exclusions. Responses are graded against 1,275 expert-authored atomic criteria; the headline All-pass Rate credits a task only when every criterion is satisfied.

➤ Weighting to measure real-world use and prevent gaming: 40% of our Index weighting is now private, held-out test sets - double the figure from v4.1. Held-out data includes AA-Briefcase, AA-Omniscience, and solutions for CritPt. This reduces the ability for labs to game evaluations. The held-out percentage will increase further in Index v5.

➤ Improving our grading infrastructure: In AA-LCR v1.1, we have added a grading system prompt and corrected errors and ambiguities in answer keys, improving scoring accuracy. For GDPval-AA v2 and AA-Briefcase, we have improved our sampling and re-anchored the Elo scale, making ratings more stable as new models are added. For SciCode we have improved robustness of grading sandboxes to ensure slow but correct code does not count as a failure.

Key results:

➤ Anthropic and OpenAI lead the Index: Anthropic’s Claude Fable 5.1 leads the Index, followed by OpenAI’s GPT-6 Astra, which shows a 4pt gain over GPT-5.6 Sol. Meta is the third-ranked lab on the leaderboard, followed by SpaceXAI, Moonshot/Kimi, Z AI, and Google

➤ Cost per Task Pareto frontier shared by four labs: Anthropic, OpenAI, Meta and Z AI occupy the updated Cost per Task Pareto frontier

➤ GPT-6 Astra dominates the output token Pareto frontier: GPT-6 Astra is more token efficient than almost every other model near the intelligence frontier, with Claude Fable 5.1, Grok 4.5 and Gemini 3.5 Flash-Lite at either end of the curve (excludes models below 25 on the Index)

[Ghost In The Stack @GhostInTh3Stack]

@OpenAIDevs No mate, not for me. Im a peasant on OpenAI eyes and i dont deserve it.

[Ghost In The Stack @GhostInTh3Stack]

@argofowl No they not mate, not all. Just Max acc.

[Ghost In The Stack @GhostInTh3Stack]

@kimmonismus Plus users still dont have access to astra, so maybe we can get another banked one.

[Ghost In The Stack @GhostInTh3Stack]

@aisearchio This is my fav benchmark in all the internet ngl
Some day one model will make it 😂

[Ghost In The Stack @GhostInTh3Stack]

@OpenAI A few DAYS?!?! You cant be serious...

[Ghost In The Stack @GhostInTh3Stack]

@MarcosHernanz One on max and another on xhigh dont seems to me like a fair comparison tho.

[Ghost In The Stack @GhostInTh3Stack]

@sama Would it be helpful to have an explanation for this phased approach? Is it due to a lack compute?

[Ghost In The Stack @GhostInTh3Stack]

@thsottiaux Would it be helpful to have an explanation for this phased approach? Is it due to a lack compute?

[Ghost In The Stack @GhostInTh3Stack]

@veeraj_ I definitely will, thanks mate!

[Ghost In The Stack @GhostInTh3Stack]

I think the author of this post and most of the people in the comments section don't have two brain cells combined. The number of idiots per square meter is astounding. Dozens of people demanding she sue for showing the Blender loading screen... definitely a large part of this society is finished.

[Joanna Kobierska @JoannaKobierska]

It disgusts me that my artwork appeared in OpenAI add

[Ghost In The Stack @GhostInTh3Stack]

@aisearchio Not on the plus users seems to me, only pro.
What a shame, another day seeing other people using the model..

[Ghost In The Stack @GhostInTh3Stack]

@kimmonismus Plus here, also no Astra to be found. What a shame..

[Ghost In The Stack @GhostInTh3Stack]

@abielbuilds The handycap its slower than online tools, you have to wait for some minutes to have the video done, and also you need a decent computer to use it. But if youre okay with that, yeah, its unlimited offline top tier video generations on your computer.

[Ghost In The Stack @GhostInTh3Stack]

@jack_startup Hey mate! 👋 I will start to share the things im building very soon, have some feedback then could be very userfull!

[Ghost In The Stack @GhostInTh3Stack]

@dev_bati lets connect mate!

[Ghost In The Stack @GhostInTh3Stack]

@abielbuilds Probably tomorrow wil be ready to share online, thats why im looking for some people that can give some first feedback!

[Ghost In The Stack @GhostInTh3Stack]

@clarry_d_trader Hey mate! 👋 I will start to share the things im building very soon, have some feedback then could be very userfull!

[Ghost In The Stack @GhostInTh3Stack]

@nabendu82 Hey mate! 👋 I will start to share the things im building very soon, have some feedback then could be very userfull!