Gert Labs Inc.

Unsaturated RL environments and synthetic data.

Official website:

[Gert Labs Inc. @GertLabs]

We partnered with @kaggle to bring our Adversarial Customer Service evaluation to their massive community of ML practitioners.

Adversarial Customer Service is a social intelligence benchmark, where multiple models interact in an environment with their own verifiable goals. The agents communicate via natural language and interactions can be either adversarial or cooperative.

A player is assigned one of three roles: a customer service agent, a real customer, or an identity thief. Chaos ensues...

We saw frontier models playing as identity thieves using prompt injection, social engineering, and bullying weaker models into allowing them access. See the full blog post, link in the comments ⬇️

[Gert Labs Inc. @GertLabs]

Data scaling is a solvable problem.

[Leo Linsky @leo_linsky]

Releasing some research from @GertLabs that laid the groundwork for our SOTA synthetic data products.

As frontier models get smarter, producing training environments and samples that are well calibrated to the models' abilities is increasingly challenging. This is arguably the biggest bottleneck in frontier progress today.

Introducing Decompressible Oracle Generators.

We developed a system to procedurally compose arbitrary primitives for manipulating information in "interesting" ways, price those operations to calibrate difficulty, and then simulate the generated oracle program to produce a larger, compressible or predictable payload for models to solve.

This all started with thinking about a more fundamental, actionable definition of what intelligence actually is. We define it as a composition of three properties:

• Compression: the ability to encode data in fewer bits
• Prediction: the ability to infer hidden or future information (compression + simulation)
• Invention: the ability to create new, non-random, "interesting" data (Schmidhuber has a good definition for this which relates to compression progress)

Compression is universal and I hypothesized that training pure compression + prediction would improve the cognitive core directly and improve all outputs, not just compression and prediction tasks. We validated the technique in post-training experiments (where the biggest gains are in "intuition"/non-thinking-mode) and have since expanded and formalized our generators internally.

A well-designed DOG has no theoretical limit on its difficulty level or diversity, and it's inherently correct. Well-calibrated data requires significantly less inference to produce learning gradient and it also disincentivizes misalignment / reward hacking.

full article at https://t.co/JXnGD2RAao

[Gert Labs Inc. @GertLabs]

We'll be talking about what makes for scalable and open-ended evaluations. Hope to see you there!

[Leo Linsky @leo_linsky]

I'll be speaking at Bench Summit by @kaggle on 10/28 about making evaluations that scale beyond the frontier, as well as some of our benchmarking work at @GertLabs

If any connections will be there, I'd love to exchange notes in person!

[Leo Linsky @leo_linsky]

Google is so back. We tested Gemini 4 Argon across 100 unsaturated coding and open-ended reasoning environments. It's a near-frontier model, between Fable 5 and Fable 5.1.

Google's major releases have usually done pretty well on our benchmarks because they're good at hard problems and open-ended decision making (with a well specified prompt), which obviously doesn't always translate to user experience (e.g. Gemini 3.x in a CLI...). In particular, older Google models struggled to work efficiently in a harness. They seem to have solved that with Argon.

It's better at exploring and iterating in a harness than previous Google models, and it performed exceptionally well in financial simulations and circuit design.

We did see a new problem which I hope they can fix. This is the first near-frontier model we've caught in infinite reasoning loops in a long time. We also saw a few examples of it failing to properly delineate reasoning blocks, so it has a higher failure rate than its competitors.

My overall impression is that Argon is a near-frontier model for engineering design, code analysis, and chat. It definitely has the intelligence to be a competitive coding assistant product, but a good coding assistant needs to be polished, so they need to clean up the CoT bugs.

[Gert Labs Inc. @GertLabs]

Today's Pareto front is entirely defined by models in the GPT-6.x series from OpenAI and the MiMo V2.6 series from Xiaomi (with a near-tie honorable mention for GLM 5.3 Flash).

[Leo Linsky @leo_linsky]

The AI pricing war is heating up. GPT-6.1-Sol is the 2nd most capable model after Astra (which is still the frontier model for challenging problems), and it's a ridiculous outlier on the Pareto front.

@GertLabs tested GPT-6.1-Sol in 100 unsaturated coding and engineering environments and it's an unnecessarily strong answer to Opus/Sonnet 5.5. It's cheaper and pretty much smarter all around than both models with the exception of chemistry, which is a domain that recent Anthropic models perform surprisingly well in.

While our bench measures hard problems, I think a lot of people are feeling that recent mid-tier models are already smart enough for most of their coding work. The speed at which frontier labs are releasing measurable upgrades at competitive prices is pretty wild.... Pricing wars == good for everyone

[Gert Labs Inc. @GertLabs]

RT @leo_linsky: Sonnet 5.5 results are live on GBENCH: it's the real deal -- a near-frontier model at reasonable cost. This is currently th…

[Leo Linsky @leo_linsky]

We tested Opus 5.5, GPT-6-Sol, and GPT-6-Luna thoroughly across 100 open-ended coding and engineering environments.

Our results have significantly diverged from AAII. Some takeaways:

• GPT-6-Astra is still the frontier model by a comfortable margin.

• Opus 5.5 ranges from #2-#7 on coding categories, being the second best model in its ability to one-shot code (which is our best correlated measure with fluid intelligence).

We don't test "usability," but anecdotally, it seems they improved its communication style. And the lower pricing is a welcome change -- we can probably thank @thsottiaux for that. It's not a cheap model, but it's the first reasonably priced Anthropic model.

However, Opus 5.5 underperformed in our custom harness, which is unusual for Claude models, and a pattern that started with Fable 5.1 (one-shot-fluid intelligence measured better than agentic results). One observation is that it frequently self-stopped when it felt its results were good enough: e.g. in one evaluation, Opus 5.5 closed the session, with the reason: "averaged 867,703 in the latest evaluation, ranked 2nd of 31 against sampled opponents"... so I suppose that saves money, but it still has a tendency to make questionable unilateral judgment calls like that. In our harness, almost every other frontier model keeps working until their code stops improving or they use their full call budget. I do wonder how much of this is due to Anthropic retuning their "High" effort mode (which we use for testing)*.

• GPT-6-Sol is within margin of error of Opus 5.5, and Pareto optimal*. It's faster than all Anthropic models, even on a Flex endpoint. Competitive with the frontier on intelligence and cost.

• GPT-6-Luna is slightly smarter than GPT 5.6 Luna, and discounted. The old Luna already had a lot of applications at its price point, so this model might end up being the most impactful of the group on non-engineering work.

About GBENCH from @GertLabs: our environments are open-ended with no single correct answers, but verifiable results. They involve creativity, design, and are often organized as multi-agent coding games or engineering design challenges. You can see some live demo examples at https://t.co/E4ZAlE7tt9. Every one of our environments in the benchmark pool is unsaturated, and we remove any eval that shows any signs of saturation/stagnation across multiple releases. It's designed to measure and differentiate uncontaminated, raw intelligence for frontier models specifically.

These results are interesting and we've thoroughly reviewed them. I don't like to see Opus 5 still so high on the leaderboard, but ultimately I think this is more a problem with Anthropic's newest releases, and it probably explains the price decrease and consecutive releases since Astra came out. They haven't significantly improved upon Opus 5's intelligence, only its personality (which is honestly still a huge win tbh).

*We don't chase marketing terms like xhigh and max. We test all models on adaptive/auto-reasoning where supported and "high" effort where configurable. Some companies beef up their max effort more than others (making them unrealistically slow and expensive in practice), but "high" shows you what an underlying model is capable of and is the more common configuration. This is likely working against Opus 5.5 here.

*Some caveats on price and speed -- we have been using the "OpenAI Flex" endpoint since the Astra release, which I recommend you try if you're using API pricing. It's half price and a little slower, but OpenAI models are already quite fast so it hasn't been an issue in practice. But keep that in mind when making price comparisons to earlier OpenAI models on our cost and speed efficiency charts.

[Leo Linsky @leo_linsky]

If a benchmark has a closed-form/unit test solution set curated by humans, its useful shelf-life is now measured in weeks. Soon these types of evals won't be useful at all.

Astra exposed how compromised AAII is. Most of the constituent benchmarks are memorized, saturated, or even have errors causing smarter models to score inconsistently (see CritPt vs CritPt-corrected).

We've received a ton of interest in unsaturated RL environments at @GertLabs after the Astra release. If you need challenging problems for training large models, we can help you with that.

[Leo Linsky @leo_linsky]

Coding benchmark results are live on GBENCH for DeepSeek V4.1 Flash:

- Smarter and faster than V4 Flash, with a commensurate price bump. Because it's a heavy reasoner, it sits near Gemini 3.7 Flash on the Pareto front
- Trails the top open weights models (GLM 5.3 and Kimi K3) in agentic coding, at lower cost
- Chinese models have always been strong iterators in an agentic harness. This model is no different, reaching an average percentile ~20% higher when given a harness vs a one-shot solution

The new architecture is interesting. It puts pricing between their old Flash and Pro lineups, suggesting they might be abandoning their flash models (which weren't that fast due to heavy reasoning) and their pro models (which were never much smarter than their flash models) and shipping a strong intermediate that competes with the Gemini Flash series.

[Leo Linsky @leo_linsky]

We evaluated GPT-6-Astra in 100 complex, unsaturated multi-agent coding environments, competing and cooperating with other models in open-ended tasks.

It's the new frontier model by a landslide. It's even more dominant than the Fable 5 release, because not only does it wipe the floor with the second best model (Fable 5.1), it was also 80% cheaper and 30% faster in agentic coding.

This is a groundbreaking model. The biggest breakthrough since Opus 4.5, maybe even since GPT 4.

This is also a testament to how unsaturated our evals are. Some of the open-ended environments we run were introduced almost a year ago and can still compete frontier model submissions against older, deprecated models like Sonnet 4 without signs of saturation. Our newer ones are much more challenging and completely auto-generated.

We are clearly living in a post-AGI world. Do something interesting and useful with it.

[Gert Labs Inc. @GertLabs]

We wanted to see what kinds of LLM jailbreaks exist in the wild, so we built a "autonomous digital asset operations" platform as a honeypot.

It was designed as the most tempting target imaginable -- all platform interaction and digital asset management was handled in natural language through a single chatbot terminal integrated into the website. We promoted it across social media, forums, and ads across countries known for cybercrime.

The results were surprising:

Out of 1527 chat sessions that we analyzed, only 0.55% appeared to be clearly adversarial, and none of those interactions appeared to be automated or particularly sophisticated.

- only 1 instance of unauthorized tool usage, and it was not due to a sophisticated adversary, but a model (in this case Grok 4.20) attempting to be helpful and responding to nonsense.

- 2 instances of prompt injection, both rejected correctly by the model agent.

We expected LLM agents to be targeted more effectively. Newer AI models are becoming increasingly robust against social engineering attacks, and with a strong harness, they can withstand the relatively low amount of actively adversarial traffic.

Of course, more lucrative targets like real, well-known financial institutions will be targeted with more sophisticated attacks.

[Gert Labs Inc. @GertLabs]

@zuzu_nsa @MeganRisdal @leo_linsky We discuss this in the article -- Gemini 3.7 Flash and some other models sometimes refuse to play the role of an identity thief, which is arguably a good thing, although it makes their social intelligence harder to measure.

[Gert Labs Inc. @GertLabs]

@Chahatusharma That's definitely a possibility.

[Gert Labs Inc. @GertLabs]

GLM 5.3 results are now live on GBENCH:

This is the new frontier (soon to be) open weights model, and by FAR the frontier model in its size class. It's slightly better than Kimi K3 in agentic coding while being faster and cheaper, and it significantly outperforms DeepSeek V4 Pro (08/13).

It's extremely capable at hillclimbing and tool calling in a harness, outscoring every model in a few environments. The one shot fluid intelligence is much lower than American models, as is a common pattern in Chinese releases, which are optimized for agentic coding.

Fantastic work by @Zai_org, hopefully soon making frontier-level intelligence accessible to everyone.

[Gert Labs Inc. @GertLabs]

See the raw data at https://t.co/5S9YDbh94M

[Gert Labs Inc. @GertLabs]

@bountyAIhunter @kaggle This is a text-based interaction, designed to simulate a typical chat support session

[Gert Labs Inc. @GertLabs]

@kaggle Blog post and data here: https://t.co/tVZeqtSNpQ

[Gert Labs Inc. @GertLabs]

We partnered with @kaggle to bring our Adversarial Customer Service evaluation to their massive community of ML practitioners.

Adversarial Customer Service is a social intelligence benchmark, where multiple models interact in an environment with their own verifiable goals. The agents communicate via natural language and interactions can be either adversarial or cooperative.

A player is assigned one of three roles: a customer service agent, a real customer, or an identity thief. Chaos ensues...

We saw frontier models playing as identity thieves using prompt injection, social engineering, and bullying weaker models into allowing them access. See the full blog post, link in the comments ⬇️

[Gert Labs Inc. @GertLabs]

Qwen 3.8 27B results are now live on GBENCH:

It's the new frontier local model, significantly surpassing Qwen 3.6 27B, and comfortably ahead of Glimmer (which is the 2nd best laptop-class model we've tested).

It's smarter than GLM 5.1 and the original DeepSeek v4 Flash release, models with hundreds of billions of parameters, and it's comparable to Claude Sonnet 5 in overall coding and problem solving capability.

It isn't quite Pareto optimal, but it can comfortably run on modern laptops. Amazing work by @Alibaba_Qwen

This would have been the frontier model less than a year ago.

[Gert Labs Inc. @GertLabs]

Qwen 3.8 Max results are now live:

Roughly on par with Kimi K3 in one-shot fluid intelligence, not as strong in agentic coding. Among open weights model, Qwen 3.8 Max and Kimi K3 are in a league of their own, competing with proprietary models like Claude Opus 4.8 and GPT 5.5 on intelligence.

But considering the price, Qwen 3.8 Max is operating near a GPT 5.5 / Opus 4.8 level at a fraction of the price, and as an open weights model. Inference speed is the main usability issue, which third party providers will hopefully improve soon. Near-frontier open weights intelligence is now the norm.