Andrew

Tokenomics @SemiAnalysis_

[Andrew @AndrewMegalaa]

@DoorDash your website is consuming more ram than my Linux VM, pls fix

[Andrew @AndrewMegalaa]

@dylan522p btw Carbon was not mentioned in the slack thread whatsoever lmao some fascinating emergent behavior here

[Andrew @AndrewMegalaa]

Checks out

[Florian Brand @xeophon]

We looked at the data and the data stared right back at us 🥀

Some insights into AutomationBench, OfficeQA Pro v2, web access in public benches and why monitoring is hard.

This has been in the work for months, more to come!

[Andrew @AndrewMegalaa]

@emollick Human baselines don't actually make sense anymore. Take some of the material science tasks for example, Astra beat the SOTA published reference on just a sub-task of one of the tasks.

On issues: we didn't release all the task material so those flags may be false positives :)

[Andrew @AndrewMegalaa]

@eliebakouch @SemiAnalysis_ Gemini has some cyber classifiers triggering refusals that antigravity retries. We are trying to work with all the labs to get unrestricted access to report no fallback / no refusal scoring

[Andrew @AndrewMegalaa]

@sujaihiremath Super cool takes, very underrepresented in the benchmarking world

[Andrew @AndrewMegalaa]

@bryancsk Was wild seeing the models solve some of those tasks!

[Andrew @AndrewMegalaa]

@michael_chomsky funny seeing this right after launching

[SemiAnalysis @SemiAnalysis_]

Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.

GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵

[Andrew @AndrewMegalaa]

Super excited to launch this with @maxkan alongside our amazing partners!

[SemiAnalysis @SemiAnalysis_]

Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.

GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵

[Andrew @AndrewMegalaa]

@PawelHuryn Read the methodology in the newsletter. It's more precise, and we own multiple accounts to check for things like this.

[Andrew @AndrewMegalaa]

Imagine having super powerful coding agents that write great @OpenTofuOrg /@PulumiCorp to spin up any infra they need on GCP and still using 15 different more expensive and less secure SaaS providers because they are "easier to use"

[Andrew @AndrewMegalaa]

Not surprising though because
1) adversaries don't have Mythos or Astra red access and contrary to popular belief, GLM 5.3 isn't actually there yet
2) frontier hackers aren't targeting your average joe, they are going for institutions and enterprises.

[Epoch AI @EpochAIResearch]

The share of US adults experiencing a cyber incident hasn't grown since Claude Fable 5's release.

About 45% reported at least one incident in the past 12 months when we asked in September, compared with 46% in June.

Other indicators have increased, such as vulnerability disclosures from notable organizations (around 5× per our previous research). Our poll with @Ipsos measures something different: whether more people experience incidents themselves.

[SemiAnalysis @SemiAnalysis_]

A $200 AI subscription can be worth $12,000 in tokens. Which plan delivers it is not close.

"They're selling to willing buyers at the fair market price. This is capitalism."
"You can pay $200 for a Claude plan and get $12,000 of Opus 5.5 tokens at API pricing. So that's obviously a good deal."
"You have to say the $200 plan, when running a particular workload on a particular model, gives you some dollar amount. But then you also have 5.5, which is a beast. $6K or $5,000 worth of value."
"And when you do that comparison, it's not even a question of who's providing more value. ChatGPT versus Anthropic subscriptions."

@maxkan and Andrew Megalaa join @JordanNanos this week

[Andrew @AndrewMegalaa]

Seems like a lot of people missed this section in our newsletter.

Anyone who used both Opus 5.5 and 6.1 Sol knows this isn't even a fair comparison as Opus 5.5 is actually competing for the Astra class in the real world.

[Tibo @thsottiaux]

The real story is this one