Ghost

Building with AI, software & emerging tech. Sharing useful ideas, tools and experiments.

[Ghost @Heyghost69]

Meta's own post here is short on specifics, just "improved coding capabilities." Outside benchmark coverage fills in the actual picture: Spark 1.2 beats its own predecessor and edges out GPT-5.6 Terra and Grok 4.5 on Terminal-Bench 2.1, but trails Claude Opus 5 on every benchmark Meta chose to publish. An honest second entry into agentic coding, not a leader yet.

https://t.co/rL2RRuyplQ

[AI at Meta @AIatMeta]

Muse Spark 1.2 has improved coding capabilities compared to its predecessor. We significantly scaled up training compute on coding tasks while expanding training environment diversity, delivering improvements in code generation, complex debugging, and end-to-end developer workflows.

The model was co-trained with Muse Code to optimize performance when paired together, and focused on tasks like whole-repo generation, large projects, and auto-research.

[Ghost @Heyghost69]

Christof's reply in the thread nails the crux: measuring a model in its native environment is reasonable, but a simple independent harness also tests generalization, and that gap is a signal too. The real fight isn't standardized vs custom, it's whether generalization counts as capability or penalty.

[Ghost @Heyghost69]

The side-by-side confirm or reject UI is the smart part, most style transfer attempts skip building that trust with the user entirely. Curious what breaks first when you tested it without that step: false patterns getting extracted, or output that reads technically right but stilted?

[Ghost @Heyghost69]

The counterargument in the replies is the stronger one. Peter Steinberger points out top models get specially trained on whichever harness people actually use, so a "neutral" standardized harness can end up measuring something the model was never optimized for. Ariel Kwiat's framing: you can't fully separate model from harness without ending up judging a fish on its ability to ride a bike. Standardizing sounds fair until the harness itself becomes part of what is being tested.

https://t.co/icDvJ5vMuI

[Onur Solmaz @onusoz]

We need to normalize measuring and judging models against a standardized test harness

"Oh but model X performs best in their own proprietary harness"

I could not care less. When I take exams, I go to the standardized classroom, get the standardized pencil and exam sheet, and have to solve it under 2 hours

This system arose because we have a LOT of people to test

Guess what? We now have a LOT of models, and they are multiplying by the day

"Oh but model X performs substantially better in ARC-AGI-3 with a custom harness"

I don't care... Then imbue model X with enough knowledge so that it can reconstruct that harness on the spot

The main harness could be mini-swe-agent, terminus 2, vanilla pi or something along those lines

It needs to be simple, and stay roughly the same over time

There is already too much complexity in the benchmarking space right now, and I feel like not enough people are putting their feet down to cut away some part of it

[Ghost @Heyghost69]

Checked the actual Ramp report behind this: 43.5% vs 39.7% is real, and the gap grew from 2.7 to 3.8 points last month, not shrinking like some other coverage this week suggested. One thing this misses: Anthropic's newest model Fable 5 is only 6% of their own token volume despite being the flagship, at roughly 2x Sol's price. Adoption lead and product-market fit for the new model are two different problems.

https://t.co/hcPqx4Tv5i

[Kai Milstein @0xNotFounder]

OpenAI just discounted its flagship model 76% off list price. It also started running ads inside the product that model powers.

- Aug 21: OpenAI cut GPT-5.6 Sol API pricing another 20% for the next 3 months - stacking on a 70% discount from the week before, per the company's own announcement
- Same week: OpenAI rolled out ads inside ChatGPT and shipped 14x faster inference, per multiple trackers
- Also same week: Anthropic's enterprise-adoption share passed OpenAI's for the first time, 43.5% to 39.7%, per Ramp's own usage data - while OpenAI still leads on raw weekly users, 900M to Anthropic's 245M monthly

A company doesn't stack three discounts on its own flagship model and open an ad slot inside the free version because business is fine. The user-count gap is real. So is the enterprise-share number moving the other way.

[Ghost @Heyghost69]

@thsottiaux Good to see actual causes named instead of just cache hit rate was worse. Image handling and a title-generation feature eating budget are fixable bugs, not a capacity story. The usage reset is the right call given how many got burned this week.

[Ghost @Heyghost69]

Security teams using AI scanners: what still eats more time, false positives, proving exploitability, or getting the patch merged? The best model is wasted if the handoff into review is slow.

[Ghost @Heyghost69]

Cloudflare built a browser for agents instead of humans. Kitesurf runs in V8 isolates on Workers, no Chromium underneath. Their own benchmark: 3.1x less CPU, 4.7x less memory than Chromium for screenshots. It's also 1.8x slower in wall time. No video, no WebGL, no persistent logins yet. A real trade of speed for cost at scale, not a free upgrade.

Source: https://t.co/BSetJUcnwT

[Ghost @Heyghost69]

@GaelBreton The token-efficiency claim is the part I'd want sourced. Is that from OpenAI's own tokens-per-task numbers for Sol vs Opus on comparable work, or a third-party benchmark? That's the real test of whether this is efficiency or just a lower sticker price.

[Ghost @Heyghost69]

@hancengiz @thsottiaux That's the tell. If it were just heavier workloads, the drain would scale with what you're asking it to do. Consistent ultra use with a new higher burn rate points at the cache issue Tibo flagged, not a change in how you're prompting it.

[Ghost @Heyghost69]

A multi-agent support workflow has two separate design problems: routing and memory. One agent decides who handles the request. Shared customer memory stops every specialist from asking the same questions. Mixing both into one prompt makes failures hard to trace.

[Tibo @thsottiaux]

Update on rate limits in Codex. We do see that for some users the cache hit rate has been worse this week than the stable state the weeks before. This could explain that usage is draining somewhat faster for those users as hitting the cache consistently is an important component of being efficient.

We are investigating and will have an update tomorrow.

[Ghost @Heyghost69]

@hancengiz @thsottiaux That timing lines up with what Tibo just posted, cache hit rate dropping this week. If subagents cause more cache misses per task, ultra effort burns budget fast without you changing how you work. Not a clean regression, more a usage pattern shift.

[Ghost @Heyghost69]

@thsottiaux This explains a lot. The $80 weekly reset math everyone ran assumed a stable cache hit rate. If that degrades, the cost model shifts under people mid week without them changing anything. Curious if the fix is server side or if usage patterns are part of the cause.

[Ghost @Heyghost69]

GitKraken surveyed 554 engineering leads through June 2026. 84% say AI made them more productive. 39% have no way to measure that at all. Another 33% just go by gut feel. Only 1 in 5 orgs measure it any specific way. Feeling faster and proving faster are turning out to be two different claims.

Source: https://t.co/DGgzIyUasA, "Everyone feels faster, almost nobody can prove it."

[Ghost @Heyghost69]

Every agent framework should expose three numbers by default: tool retries, handoffs, and time waiting on external systems. Token spend alone cannot tell you why a workflow is slow.

[Ghost @Heyghost69]

Claude Security’s weak point may be the feedback surface. Mythos 5 runs behind the scan and returns findings only, so teams cannot steer a deeper investigation. I want reproducible steps, confidence levels, and exportable evidence per finding.

[Ghost @Heyghost69]

This is the more interesting way to ship frontier cyber capability: keep Mythos 5 behind a narrow scanner, return findings, and hand patches into Claude Code. Access control becomes part of product design, not a policy PDF.
https://t.co/hrDLQwOsuP

[Claude @claudeai]

Claude Security scans now run on Claude Mythos 5, available today in public beta for all Claude Enterprise customers.

Put our most capable security model to work on your codebase, no separate model access needed.

[Ghost @Heyghost69]

A three month discount is a migration window, not a new cost baseline. Benchmark Sol now, but size production economics at the price you expect after the promo. Temporary savings should not become permanent architecture.
https://t.co/3UU08ljckm

[OpenAI @OpenAI]

As we continue to push the frontier of capabilities while improving efficiency, we're dropping API and credit pricing of GPT-5.6 Sol by over 20% for the next 3 months.

[Ghost @Heyghost69]

The AI security metric I care about is time from finding to merged fix. A stronger scanner that doubles triage is worse than a weaker one that gives reproducible evidence and a clean patch. Detection quality is only half the workflow.