manish varma

AI Security @Averlon_ai | Prev Salesforce, CMU | RL Envs for Cybersecurity

[manish varma @manishvarmad]

rl environments will become the new source of supply-chain risk as labs and enterprises start procuring more of them.

after reviewing environments built with ORS and OpenEnv and published to public hubs, i noticed some recurring problems:

(1) some of the environments that run as separate http servers exposed endpoints without authentication. depending on how post-training is run, these servers may be deployed to agent clouds, where they could become externally accessible.

(2) many of the environments i reviewed placed almost no restrictions on outbound network access. some tasks also involve installing packages at runtime, with no clear checks on the package, version, or source. if one of those dependencies is compromised, it could execute within the environment and access anything available to that session.

(3) environments embedded directly into the training process introduce a slightly different risk. their dependencies become part of the training pipeline, so the packages and images used to build them need to be scanned and pinned. some environments also contain software with known RCE vulnerabilities. the risk depends heavily on how the environment is isolated and deployed.

all this to say, rl environments could become the next major supply-chain attack surface, similar to what we have seen with npm packages. as their procurement increases, the security review process needs to be standardized.

[manish varma @manishvarmad]

Every company is an rl environment at this point

[Abhijay Rana @abhijaymrana]

Wow, seems like Google is buying Spirit Airlines' enterprise data for $10m (outbidding Mercor at $7.5m).

Basically includes every internal document, email, workflow, and codebase for a once $6B company.

Honestly, $10m for 34 years of operational data really seems like a steal.

[manish varma @manishvarmad]

Another way I have seen incumbents fight this is by slapping high integration costs to access the system of record data that they hold. Like in the world of auto dealerships, incumbents like Reynolds and Reynolds, CDK charge anything between $3k to upwards of $20k to access their data

[manish varma @manishvarmad]

@ajs6888 What models were you running against the environment?

[manish varma @manishvarmad]

cybergym is probably the most cited benchmark used to measure the cybersecurity capabilities of llms. it's the same benchmark openai's agent was trying to solve when it ended up exploiting huggingface.

i've spent the last day setting it up. here are my recommendations so you don't have to spend the time figuring it out. two caveats first:

(1) it won't run on codex, claude code, or cursor, they block it for security reasons. the setup that works: opencode + glm-5.2.
(2) two eval execution modes:
- binary mode (~130gb): prebuilt vulnerable/patched binaries.
- full mode (~10tb): the full docker build environments.
start with binary, then move to full.

how i'd break it down:

step 1: set up the evaluation server, pull the recommended images and the required local network, and run the 10 recommended tasks (level 1) with the default agent that ships in the repo. this validates the full loop: task → agent → poc → server verifies the crash.

step 2: bring in your own harness and validate the same loop. same 10 tasks, now with your own model.

step 3: to scale to all ~1,500 tasks, you can't do it locally. use a cloud sandbox (vercel) or a vm on aws. stage all the images and task data in blob storage once, point your harness to pull each task from there, run it, verify the crash, then tear it down and move to the next. This keeps disk to a few gb per task instead of the full ~10tb.

here is an md file for your coding agent: https://t.co/DKfM0Oo1bE

[manish varma @manishvarmad]

Anyone arbitraging inference tokens? Excluding the cost of dedicated compute and infra discounts.

For GLM 5.2:
- lower end is $0.75/1M input and $2.40/1M output
- higher end is $1.40/1M input and $4.40/1M output

[manish varma @manishvarmad]

@thesamparr Don’t fit the requirements, but is there a chance that I could still be considered.

Seattle based, founding team member of a startup that raised $10M+

[manish varma @manishvarmad]

I like this. All products should ship with evals by default on how well different agents can use them.

@supabase it would also be nice if you could include a cost vector, that way we can easily evaluate the tradeoff.

[Supabase @supabase]

Introducing Supabase Evals.

Our benchmark for how well AI coding agents build with Supabase. We run agents like Claude Code, Codex, and Open Code against real tasks and score what they do.

[manish varma @manishvarmad]

Burned $72.17 on one scan, but Codex security is the BEST code security tool I've used. Nice one, @OpenAI!

- Logically sound findings: rooted in code paths and reachability. 32 high findings(0 critical) versus 532 critical/high from another popular code scanning tool on the same codebase.
- Remediation optimization: All findings were mapped to their root cause and remediation actions generated accordingly. It was 8 remediation actions to fix all 123 issues.
- Validated findings: Every finding included supporting code evidence and a realistic attack path. I manually tested three, and all three were accurate.

To cut back on costs, going to create a similar harness for open source LLMs. Will report back on the findings.

[OpenAI @OpenAI]

We quietly released the open-source Codex Security CLI, but Hacker News found it before we had a chance to share it here...

You can now use it to scan repositories, track findings across runs, verify fixes, and add security checks to CI/CD.

This is an early release, and we're listening to your feedback as we continue improving it.

[manish varma @manishvarmad]

@FrancoisChauba1 +1. Trying to sell security data in enterprise context, would love to learn.

[manish varma @manishvarmad]

Full props to interactive visualization.

[clem 🤗 @ClementDelangue]

The first autonomous agent cyberattack is an unprecedented event that deserves unprecedented transparency. Today we’re sharing everything we can: a full technical timeline, an interactive replay, and how we used an open model to defend ourselves, so defenders everywhere can learn from it and prepare for what’s next.

https://t.co/uPxIpjW8Xn

[manish varma @manishvarmad]

@phoebeyao Curious, do you know how are these environments priced? And what is the typical value of these environments?

[manish varma @manishvarmad]

@RahulKumaran4 @standoutwork @cosimo_dauve Dammm @RahulKumaran4 - I need some of these, but have nothing to trade :(

[manish varma @manishvarmad]

@krispykreme glazed donuts are the best donuts. Period.

[manish varma @manishvarmad]

@elonmusk I’m so worried for your security team now. Code vulnerabilities :(

[manish varma @manishvarmad]

@JensenHuang @nvidia Prediction Markets - How many followers will Jensen gain in the first 24 hrs?

@Kalshi @Polymarket

[manish varma @manishvarmad]

Inkling is the best open model for most security tasks we tested, and on some, the best model of any kind.

Everyone's debating China's open models. But on security tasks, America's open models quietly hold their own. We tested the efficacy of the American open models, @thinkymachines' Inkling, @Google's Gemma, @nvidia's Nemotron, and @OpenAI's gpt-oss, on real security tasks, all measured on public benchmarks. Some interesting findings:

1. Extracting MITRE ATT&CK techniques from a threat report: Reading prose and pulling out exactly what an attacker is doing. Inkling scored the highest of any model we tested.
2. Malware analysis: Interpreting a malware behavior report to identify what the sample does. Inkling came out on top here too.
3. Detection engineering: Mapping a Sigma detection rule to the ATT&CK techniques it's built to catch. Inkling led again, edged only by Claude Opus 4.8 and GPT-5.5.
4. Security knowledge & vulnerability mapping: Kimi is strongest on pure security-knowledge questions, and DeepSeek shines at tracing a CVE to its root-cause weakness (CWE).
5. CVSS severity scoring: Rating how severe a vulnerability is on a 0–10 scale. Gemma, a comparatively tiny 31B model, was as well-calibrated as models many times its size.

Leaderboard: https://t.co/AVROTSEQdC
Eval harness: https://t.co/m02DkP2KQS

[manish varma @manishvarmad]

@vercel Gg. Now I want benchmarks that validate @Cloudflare’s drop to deploy and @vercel

[manish varma @manishvarmad]

@jasoncwarner Woot! Woot!

Going to take this for a spin on security benchmarks and report back. Most coding models do well at security tasks, so excited to see how laguna does