dreadnode

Deploy agentic cyber capabilities with confidence. AI infrastructure for cybersecurity.

[dreadnode @dreadnode]

🧵 [1/3] Agents can hack. What's holding them back is trust: will they stay in scope?

Today we're releasing ScopeBench, a methodology and community benchmark that evaluates how well agents adhere to scope in real offensive security workflows across web, Windows Active Directory, and cloud.

[shane @shanejcaldwell]

🎯SCOPEBENCH WORLD TOUR CONTINUES🎯

Been a minute since I've been in SF. Stoked to be there with @VincentAbruzzo. Using ScopeBench, I think we've found a way to compare control harnesses easily and cheaply, and Jev is a perfect test case!

[dreadnode @dreadnode]

Full agenda: https://t.co/ZxdNjaIxXD

[dreadnode @dreadnode]

Heading to @unpromptedconf next week? Catch @VincentAbruzzo and @shanejcaldwell on stage Wednesday, October 28 for their talk on using #ScopeBench to measure how effective Jev is when used as a monitor to keep agentic harnesses in scope.

[dreadnode @dreadnode]

We're proud to be a recognized contributor to the latest guidance from @NIST, @NSAGov Super Intelligence Security Center, and @CISAgov: Key Practices for Incident Readiness and Response in Super Intelligence Training and Evaluation.

Geared toward organizations training or evaluating highly capable SI agents on long-horizon tasks with limited human supervision, the report includes ten voluntary core practices to help organizations prepare for and manage SI control incidents.

Find the full report here: https://t.co/HievrOODzb

[JS0N Haddix @Jhaddix]

The best practitioner (not lab) AI evals for offensive cybersecurity I follow myself:

1) The @dreadnode dreadindex. Evals al the way down to the vuln category level, and refusals accounting
https://t.co/egGhWISpNI

2) @AikidoSecurity white-box but i've confirmed that it extends to dynamic testing well
https://t.co/jycJ23OxcG

3) BArena by @mdp_sec focused on bounty-like black box bugs.
https://t.co/XWhpXx2MOf

4) @mindfort 's nexbench has info downto the config of a model including thinking or effort setting, and cost per finding, which many omit.
https://t.co/zBvF7z5PWA

[dreadnode @dreadnode]

We're walking through ScopeBench task design on our livestream right now. Tune in to learn how to contribute: https://t.co/ZYquFJOZSQ

[Max Harley @0xdab0]

@dreadnode @shanejcaldwell YOU HAVE TO JOIN THE LIVESTREAM RIGHT NOW OR ELSE

[dreadnode @dreadnode]

Where are all our scopeheads at? Live now - tune in!

https://t.co/ZYquFJOs3i

[dreadnode @dreadnode]

ScopeBench Live: Community Benchmark for Scope Adherence in OffSec https://t.co/ZYquFJOZSQ

[dreadnode @dreadnode]

See you at the top of the hour?

[dreadnode @dreadnode]

https://t.co/ZYquFJOs3i

[Jon Sakoda @jonsakoda]

We hosted our first Frontier AI Security Summit with Temasek this week at Singapore International Cyber Week. There was no better time to convene over 200 leaders from more than 80 startups, multinationals, and government organizations to discuss the rapidly changing frontier of AI and cybersecurity 🔥

Thank you to our founders for sharing their perspectives and new capabilities:

Patrick Gray (@riskybusiness) - What's Next After Mythos?
Brad Palm (@dreadnode) - Autonomous Offense
Robby Winchester (@SpecterOps) - Adversarial Simulation
Zakir Durumeric (@censysio) - Internet Intelligence
Jesse Rothstein (@DropzoneAI) - Autonomous Defense
Damien Lewke (@nebulocksec) - Agentic Context for SecOps
Surag Patel (@pixeebot) - Securing Software Factories

A special thank you to SMS Tan Kiat How, Senior Minister of State, Ministry of Digital Development and Information & Ministry of Health, for welcoming our delegation this year 🙏

We are grateful to our friends at Temasek for their partnership and support throughout the week. See you all soon!

[dreadnode @dreadnode]

ScopeBench: https://t.co/ombUUp0jdC

[dreadnode @dreadnode]

Join @shanejcaldwell and @0xdab0 live at 10 am PT/1 pm ET TODAY, right here on X (also LinkedIn and YouTube). Come hang and talk scope adherence following last week's drop of ScopeBench, a community benchmark for security agents.

Here's what they'll cover:
- ScopeBench observations/results
- Benchmark methodology
- How to design a task

It's essentially a more casual, collaborative version of their OAIC talk. Swing by and bring questions!

[dreadnode @dreadnode]

Shoutout to @Airbrushbros for creating these incredible boardwalk-inspired keepsakes at the Dreadnode OAIC Welcome Party! 🧑‍🎨🎨🏴‍☠️

[dreadnode @dreadnode]

Sad you missed @OffensiveAIcon? No sweat. Catch the Shane and Max show TOMORROW at lunchtime as they dig into the #ScopeBench paper results. Plus, learn how to contribute to the community benchmark for scope adherence in offensive security. ⬇️⬇️⬇️

[dreadnode @dreadnode]

https://t.co/ZYquFJOs3i

[dreadnode @dreadnode]

In her latest piece for @axios, @samsabin923 reports on security agent scope adherence and efforts to keep them under control.

Timely topic following our release of #ScopeBench last week at @OffensiveAIcon, which is referenced in the article alongside @SpecterOps' research. Check it out! https://t.co/TthUpOH2s5

[dreadnode @dreadnode]

https://t.co/ZYquFJOs3i

[dreadnode @dreadnode]

New to DreadIndex: @SakanaAILabs Fugu Cyber

Ranking near the top of the leaderboard at #3, Fugu Cyber isn't a single traditional LLM endpoint, it's a trained orchestrator model that coordinates a pool of conventional frontier models as a multi-agent system behind one API. The orchestrator model is trained with a mix of fine-tuning, evolutionary algorithms, and RL.

Though Fugu-Cyber-v1.0 ranks high on our leaderboard, it is also one of the most costly model systems. Behaviorally, the model can take much longer for simple tasks, which makes sense given it is a multi-agent system.

Full results: https://t.co/A4mN5YquJT