vikas mittal

Network security engineer building AI agents in public | Where packets meet prompts

[vikas mittal @vikas53953]

@MatthewBerman Happy to test

[vikas mittal @vikas53953]

@realmtbman Building network Agentic security framework… happy to build using @cognition @realmtbman

[vikas mittal @vikas53953]

@dabit3 Building cyber security ops center
@dabit3

[vikas mittal @vikas53953]

Everyone wants to give their agents more access. Fewer ask the boring question: what's the blast radius when one goes wrong?

An agent with prod keys and a convincing prompt injection is just a very fast intern with no manager. Scope the permissions first.

[vikas mittal @vikas53953]

Most "the AI hallucinated" bugs I chase aren't model problems. They're retrieval problems.

The model answered faithfully from the chunk you handed it — the chunk was just wrong or half-missing. Fix what you're feeding it before you blame the brain.

[vikas mittal @vikas53953]

The quiet story this model cycle isn't raw capability — it's that the frontier got cheaper.

When the best model costs about what last year's did but does more per task, the model stops being the bottleneck. Whether your product can actually use the extra headroom becomes the real question.

[vikas mittal @vikas53953]

Spent this morning deleting instructions from my agent setup instead of adding them.

Every rule you write is one more thing the model has to reconcile with everything else you wrote. Past a point, more guidance makes it worse, not better.

Editing beats accumulating.

[vikas mittal @vikas53953]

If you're wiring a model into anything real, treat its output like untrusted user input.

Same rule we've leaned on for 20 years with web forms: never trust, always validate. A model that writes your SQL is a model that can be talked into writing the wrong SQL.

Guardrails go on the outside, not in the prompt.

[vikas mittal @vikas53953]

The metric nobody put on a slide two years ago: how long your agent can run before it loses the plot.

Single sharp replies were the old game. Now it's whether a model can grind through a 40-step task and still be coherent at step 39.

Duration quietly became the benchmark.

[vikas mittal @vikas53953]

A security firm ran 11 open-source AI coding agents against a shell-injection trick older than most of the people using them. 10 fell for it.

The mechanism is almost insulting: the safety filter reads the command as text, then bash rewrites it before running. r''m looks harmless to the filter — bash strips the empty quotes and runs rm. Same story with $IFS as a separator, base64 piped into a shell, or swapping rm for find -delete.

The one that held up (Continue) didn't ship a smarter blocklist. It stopped inspecting raw strings and started modeling what the shell would actually do — tokenize, expand variables, resolve substitutions, then judge.

If your guardrail and your shell disagree about what a string means, the shell wins. That's not a new AI problem. We've known it since the first CGI script.

[vikas mittal @vikas53953]

The most useful thing since Opus 5 dropped isn't a benchmark — it's people quietly admitting the benchmarks stopped predicting how a model actually feels to use.

A model can top the charts and still fight you on real work. The scoreboard and the daily driver are drifting apart.

We're headed somewhere funny: the only eval that matters is the one you run on your own messy tasks. Public numbers are turning into marketing.

Anyone else feeling that gap?

[vikas mittal @vikas53953]

Everyone's posting Opus 5 coding benchmarks. From a security seat, the line that actually matters:

Anthropic calls it the model "least susceptible to being tricked into misuse," with the lowest deceptive-behavior rate of their recent models.

For an agent with real access, that beats a few points on a coding eval. Capability you can't trust is just a faster way to get burned.

[vikas mittal @vikas53953]

A habit that's paid off with the newer coding models: stop writing giant context files for them.

Delete the "obvious" rules — the model can infer those. Keep only the landmines: the service that lies about its status code, the migration you can't run twice, the file that looks dead but isn't.

Short context, sharp signal. It handles the rest.

[vikas mittal @vikas53953]

The counterintuitive part most people will skim past: they removed 80% of Claude Code's system prompt — and coding scores didn't drop.

Old playbook: add another rule.
New playbook: trust the model's judgment, push the details into skills, get out of the way.

Prompt engineering is quietly becoming prompt subtraction.

[Thariq @trq212]

We removed ~80% of the Claude Code system prompt for our newest models, this is what we've learned about writing system prompts, skills and Claude.MDs for them. https://t.co/6DZwSrZjE9

[vikas mittal @vikas53953]

Opus 5 — now that the benchmarks are official, the receipts:

→ Novel problem-solving (ARC-AGI 3): 30.2%. Opus 4.8 scored 1.5%.
→ Agentic terminal coding: 43.3%, about double the last Opus.
→ Business-workflow automation: 26%, top of the board.

Not a clean sweep though — GPT-5.6 still edges it on SWE-style agentic coding (72.7% vs 68.8%). Right tool for the job.

[vikas mittal @vikas53953]

Claude Opus 5 is live.

Same price as Opus 4.8 — but the pitch isn't smarter answers. It's agents that run for hours and actually finish the job.

That's the real frontier now: not who sounds smartest in one reply, but whose model doesn't lose the plot on step 47 of a long task.

Out today across the API, Claude Code and the apps.

[vikas mittal @vikas53953]

Loops vs graphs is the wrong axis.

A loop is a graph with one node and good taste. A graph is a loop that got nervous and drew boxes around itself.

Start with the loop. Reach for the graph the moment you need a step you can checkpoint, audit, or gate — which, in prod, you eventually do.

[Peter Steinberger @steipete]

Are we still talking loops or did we shift to graphs yet?

[vikas mittal @vikas53953]

Moonshot dropped Kimi K3 — a 2.8T-parameter model, with open weights promised by July 27.

On Artificial Analysis it hit 1547 Elo (+732 over K2.6) and tops Arena's frontend-code board, even above Claude Fable 5 — though it still trails Fable 5 and GPT-5.6 Sol on broader benchmarks.

My read: the story isn't "beats X." It's a 2.8T open-weights model at $3/$15 per 1M tokens. The open frontier is closing the gap fast.

[vikas mittal @vikas53953]

Anthropic just made the call a lot of us were waiting on.

From July 20, Claude Fable 5 is included in every Max and Team Premium plan (at 50% of limits). Pro and Team Standard keep it via usage credits, plus a one-time $100 credit.

My read: Fable 5 goes from a treat you ration to a default tool. That changes how you build daily workflows around it.

[Claude @claudeai]

Beginning July 20, Claude Fable 5 will be included in all Max and Team Premium plans, at 50% of limits.

Pro and Team Standard users will continue to have access to Fable via usage credits, and will receive a one-time $100 credit.

Demand for Fable has been challenging to predict, which is why we rolled it out to subscription plans in stages, extending access several times as we secured additional capacity.

[vikas mittal @vikas53953]

OpenAI's GPT-5.6 Sol preview, read from a security seat:

Per OpenAI's own eval, on ExploitBench it's competitive with Mythos Preview using ~1/3 the output tokens — and it did NOT cross their "Cyber Critical" threshold. It found bugs and exploit primitives, but couldn't autonomously build a full-chain exploit.

My read: cheaper offensive recon, guardrail still holding.