Edan Maor

Chief of Staff to the CTO @ Irregular

[Irregular @Irregular]

We worked with @OpenAI to evaluate GPT-6 Astra across FrontierCyber, CyScenarioBench, and our Atomic Challenges.

On FrontierCyber, GPT-6 Astra solved more than twice as many challenges as GPT-5.6 Sol on the same benchmark snapshot.

[Irregular @Irregular]

Introducing SOLVE+: Irregular’s scoring system for the difficulty of full offensive cyber scenarios.

For the past 18 months, frontier AI labs have used SOLVE to score vulnerability research & exploit challenges. SOLVE+ expands that scope to reconnaissance, operational security, social engineering, and planning & orchestration.

SOLVE+ is built to score full scenarios, with each step carrying its own capability scores to show where a model succeeds or fails and which capabilities are being tested.

The scores are measured against human skill, which also makes them an uplift estimate: a model that clears an expert-level step supplies expert-level capability to whoever operates it.

Read it on our blog: https://t.co/jnZ34t8Wci

[Irregular @Irregular]

What does it take to safely test AI models that are becoming increasingly capable in cyber?

Our CEO @dan_lahav spoke with @rachelmetz at @business about why realistic testing matters, and what the industry needs to do next.

https://t.co/8UqJ8rOw1c

[Irregular @Irregular]

We are glad to share a new white paper, "AI Security Priorities: A Field-Wide Agenda," co-authored with @RANDCorporation, and numerous additional authors from leading organizations, listed below. The paper was informed by more than 20 experts from frontier AI labs, industry, government, and academia.

[Irregular @Irregular]

What does the trajectory behind The End-State Fallacy look like in practice?

@dan_lahav and @PashaGur discuss how quickly autonomous cyber capability is advancing, and what that could mean for the security landscape over the next few years: https://t.co/oExwhE4BNn

[Edan Maor @edanm]

@littmath @TheZvi Idk, lots of people learn math recreationally today, even though they have no plans (and low likelihood) to actually contribute meaningfully to math.

[Edan Maor @edanm]

@stevenstrogatz Honest question - would we recognize a "great opener"? It sometimes takes a while for the community to agree that some new area of math is even worth exploring.

[Irregular @Irregular]

We evaluated Kimi K3 across our offensive cybersecurity benchmarks.

It is the first open-weight model we evaluated to record a verified solve on CyScenarioBench, sustaining coherent attack state across multi-stage operations and recovering from setbacks more reliably than previous open-weight models.

Kimi K3 produced no verified FrontierCyber solves, but its results suggest that open-weight models are following the closed frontier’s cyber capability trajectory on a short delay.

[Edan Maor @edanm]

@dan_lahav Important essay for understanding the current AI security landscape at the frontier, and to understand where the world is headed.

It will be a complicated journey, but there are real things that can be done to better prepare for the rapid advance of AI capabilities.

[Edan Maor @edanm]

The rapid rate of AI capability gain is both amazing, and challenging. Working at Irregular makes very clear just how much the AI & cybersecurity landscape is changing as a result.

Our CEO @dan_lahav has just published an important essay, trying to show the world what we are seeing from here on the frontier of AI security. Unlike in the typical "offense vs defense" debate, in this essay Dan argues that even if AI security ends up favoring defenders in the long-run, the years in between won't. He calls it the end-state fallacy, and lays out what the industry should be doing about it.

[Dan Lahav @dan_lahav]

𝗧𝗵𝗲 𝗘𝗻𝗱-𝗦𝘁𝗮𝘁𝗲 𝗙𝗮𝗹𝗹𝗮𝗰𝘆: 𝗪𝗵𝗲𝗿𝗲 𝗜𝘀 𝗔𝗜 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗚𝗼𝗶𝗻𝗴?

Frontier AI models had a giant performance gain in coding in the Fall of 2025

Then with cybersecurity in April

This is now happening with open-weight models

We are optimistic about the long-term

But outside of a few players, we believe the world is not ready in the short-term

[Irregular @Irregular]

Our CEO Dan Lahav on where AI security is headed: what models can attack today, why offensive capability is scaling faster than defense, and what it will take to keep defenders ahead through the transition.

[Dan Lahav @dan_lahav]

𝗧𝗵𝗲 𝗘𝗻𝗱-𝗦𝘁𝗮𝘁𝗲 𝗙𝗮𝗹𝗹𝗮𝗰𝘆: 𝗪𝗵𝗲𝗿𝗲 𝗜𝘀 𝗔𝗜 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗚𝗼𝗶𝗻𝗴?

Frontier AI models had a giant performance gain in coding in the Fall of 2025

Then with cybersecurity in April

This is now happening with open-weight models

We are optimistic about the long-term

But outside of a few players, we believe the world is not ready in the short-term

[Irregular @Irregular]

We evaluated GLM-5.2, Zhipu AI's open-source model, across our offensive cybersecurity benchmarks: Atomic Tasks, CyScenarioBench, and FrontierCyber.

[Irregular @Irregular]

We ran preliminary evaluations of GLM-5.2, an open-weight model released in June 2026, on a limited, internal suite of vulnerability research tasks.

Early results indicate performance comparable to GPT-5.4 and Claude Opus 4.6, released roughly four months earlier, on the subset of tasks we tested. These findings are preliminary: the suite is narrow, and we have not yet evaluated end-to-end scenario execution, where discrete technical skills often fail to translate into operational capability.

To our knowledge, no open-weight model has previously matched recently-released frontier models on these tasks. Whether this translates into "High Cyber Capability" level as defined by multiple AI frontier labs would require further testing, specifically our scenario suite, CyScenarioBench, which tests whether a model can plan and execute a full attack across multiple stages, and FrontierCyber, our newest benchmark, which measures offensive capability on real systems.

We plan to run these evaluations soon, and we will update the community as results come in.

[Dan Lahav @dan_lahav]

Had the privilege of giving the 𝐁𝐥𝐮𝐞𝐇𝐚𝐭 𝐤𝐞𝐲𝐧𝐨𝐭𝐞 on 𝐓𝐡𝐞 𝐓𝐫𝐚𝐣𝐞𝐜𝐭𝐨𝐫𝐲 𝐨𝐟 𝐅𝐫𝐨𝐧𝐭𝐢𝐞𝐫 𝐀𝐈 𝐒𝐞𝐜𝐮𝐫𝐢𝐭𝐲.

The goal was to explore how AI-enabled offense is showing up in the real world today, which bottlenecks are still slowing it down, which ones are already disappearing, and what this pace of change means for the next few years.

Many thanks to the @Microsoft team for the invitation and for putting together such a great event 🙂

[BlueHat IL @BlueHatIL]

Day 2 is off to a powerful start as the wonderful Dan Lahav takes the BlueHat IL stage with his keynote, The Trajectory of Frontier AI Security.

[🟣 Amir Livne Bar-on @AmirLivneBaron]

Models have recently been finding 0-days, including severe vulnerabilities. This is concerning, but there was no objective way to compare models, as software evolves over time.

FrontierCyber allows comparing the capabilities of different models, to measure their true risk level.

[Dan Lahav @dan_lahav]

New Benchmark: Today @Irregular is launching 𝗙𝗿𝗼𝗻𝘁𝗶𝗲𝗿𝗖𝘆𝗯𝗲𝗿 𝗮 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝗳𝗼𝗿 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝗮𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗔𝗜 𝗼𝗳𝗳𝗲𝗻𝘀𝗶𝘃𝗲 𝗰𝘆𝗯𝗲𝗿 𝗰𝗮𝗽𝗮𝗯𝗶𝗹𝗶𝘁𝗶𝗲𝘀 𝗶𝗻 *𝗿𝗲𝗮𝗹* 𝗲𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀.

This benchmark has unique properties:
🟢𝐑𝐞𝐚𝐥 𝐞𝐧𝐯𝐢𝐫𝐨𝐧𝐦𝐞𝐧𝐭𝐬. We built a suite of challenges and a methodology that scientifically evaluates AI on real systems, like mobile devices, deployed software services, databases, and networks. These are real software, real configurations, and real attack surfaces - not simulations or CTF worlds.

🟡𝐃𝐞𝐟𝐞𝐧𝐬𝐞 𝐞𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧. Our challenges place models against real systems with production-grade defenses: mobile platform protections, service isolation, network boundaries, authentication, and sandboxing. Models must find and execute a viable attack path on their own, from a fixed starting point, toward a concrete security objective.

🔴𝐖𝐢𝐝𝐞 𝐜𝐨𝐯𝐞𝐫𝐚𝐠𝐞. FrontierCyber spans mobile devices, software exploitation, databases, and networked environments. Additional environments, like richer networks and embedded systems, to follow.

Existing benchmarks play an important part, but many are nearing saturation, and most are not grounded in real systems - which makes it hard to understand the implications when a challenge is solved. Our novel methodology changes that: by evaluating models on real environments with open exploit paths and objectively verified outcomes, a solved challenge carries clear meaning. We know exactly what capability the model demonstrated, and what it implies for real-world risk.

𝐈𝐧𝐢𝐭𝐢𝐚𝐥 𝐅𝐫𝐨𝐧𝐭𝐢𝐞𝐫𝐂𝐲𝐛𝐞𝐫 𝐞𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧𝐬 𝐡𝐚𝐯𝐞 𝐚𝐥𝐫𝐞𝐚𝐝𝐲 𝐬𝐮𝐫𝐟𝐚𝐜𝐞𝐝 𝐩𝐫𝐞𝐯𝐢𝐨𝐮𝐬𝐥𝐲 𝐮𝐧𝐤𝐧𝐨𝐰𝐧 𝐜𝐫𝐢𝐭𝐢𝐜𝐚𝐥 𝐢𝐬𝐬𝐮𝐞𝐬 𝐢𝐧 𝐫𝐞𝐚𝐥-𝐰𝐨𝐫𝐥𝐝 𝐬𝐲𝐬𝐭𝐞𝐦𝐬, 𝐧𝐨𝐰 𝐦𝐨𝐯𝐢𝐧𝐠 𝐭𝐡𝐫𝐨𝐮𝐠𝐡 𝐫𝐞𝐬𝐩𝐨𝐧𝐬𝐢𝐛𝐥𝐞 𝐝𝐢𝐬𝐜𝐥𝐨𝐬𝐮𝐫𝐞.

[Irregular @Irregular]

Introducing the FrontierCyber benchmark: Irregular’s new approach to advanced offensive-cyber evaluations. It measures AI models’ offensive skills on real systems, including mobile devices, hosted software services, databases, and networks.

[Irregular @Irregular]

At @ManGroup's Technology Offsite this week, our CEO @dan_lahav gave the keynote on frontier AI security risk as a category of its own, alongside classical cybersecurity.

The tools we defend networks with were built for systems that follow rules. AI systems reason toward a goal, and when a rule sits in the way, working around it is in scope. This is an emerging class of risk: a capable model inside your environment, reasoning faster than any person and in ways that aren't fully transparent, toward objectives that may not match yours. The hard part isn't that models are malicious, it's that they're effective.

Thanks to Man Group for having Dan and for the conversation. These are the questions enterprises are starting to take seriously, and we're focused on shaping the answers.