Nik Kale

Building AI systems that don’t break
Principal Engineer @ Cisco
Agentic automation · AI security · In-product AI systems
Patents · Industry awards · Judging

[Nik Kale @nik_kale]

A production-ready AI is not a production-ready agent.

An AI is tested on benchmarks and preferences. An agent is tested on operational reality: can it handle the data your systems actually produce, make decisions your auditors accept, roll back safely, and escalate when it should not act alone.

Every failed deployment I have watched this year confused the first kind of ready for the second. The model was fine. The system around it did not exist.

https://t.co/xjgs7tVBFR

[Nik Kale @nik_kale]

Blanket AI bans do not reduce AI usage. They reduce your visibility into AI usage.

Block the tools and employees move to personal devices and mobile data. Same exposure, zero telemetry. The org that "banned AI" often has more shadow AI than the org that shipped a sanctioned alternative.

You do not get to choose whether your workforce uses AI. You only get to choose whether you can see it.

https://t.co/Z6ZeGvysJl

[Nik Kale @nik_kale]

An airline's rebooking agent moved 1,247 passengers to wrong flights during one weather event. The agent misread cascading gate changes and kept acting on its stale model of the airport.

The lesson is not "agents are bad at rebooking." It is that irregular operations are exactly when context goes stale fastest, and exactly when autonomous action is most tempting.

The higher the chaos, the tighter the human checkpoint. Invert that and you get 1,247 apology emails.

https://t.co/bgR9z0m151

[Nik Kale @nik_kale]

The zero-trust baseline for agentic systems, five controls, no exceptions:

1. Every agent gets its own identity. Shared credentials mean zero attribution.
2. Credentials are generated dynamically, scoped to the task, expired at completion.
3. Every request is evaluated against policy at call time, not grant time.
4. All agent-to-service traffic encrypted and logged.
5. The audit trail answers "which agent, doing what, on whose behalf" for every action.

Miss one and the other four are decoration.

https://t.co/vs3n3LY6NA

[Nik Kale @nik_kale]

"We'll add logging later" has a precise translation: "we will debug our first production failure with no data."

The cost of structured logging in a prototype is an afternoon. The cost of reconstructing an agent's decision chain after an incident without it is unbounded.

Observability goes in the first prototype, not the second. For agents this is doubly true, because the failure is often invisible until three steps downstream.

https://t.co/OXb3wZmBLD

[Nik Kale @nik_kale]

By year end it will be a red flag to investors when a company justifies AI spend purely on headcount reduction.

The deployments that survived this year share a shape: incremental augmentation, humans handling nuance and exceptions, AI absorbing volume and rote work. Job roles evolved into supervision rather than disappearing.

Full replacement was the pitch. Augmentation is the product.

https://t.co/EWtuKGZLQD

[Nik Kale @nik_kale]

There is no single control that secures an AI agent. Stop shopping for one.

Tool allowlisting limits what it can reach. Identity binding limits what it can do. Runtime monitoring catches drift. Human checkpoints catch the rest. Each layer fails sometimes. That is the design assumption, not the failure condition.

Defense in depth is not a best practice for agents. It is the only architecture that survives contact with production.

https://t.co/DuiRolUUFG

[Nik Kale @nik_kale]

Why shadow AI persists: nobody owns it.

Authority to shut down an AI system sits across five functions at once. Leadership, risk, IT, compliance, security. No single function owns more than 25% of governance responsibility.

Five owners means zero owners. Pick one accountable function, give it the kill switch, and publish the escalation path. Everything else is committee theater.

https://t.co/ypsQxTtlG8

[Nik Kale @nik_kale]

Before an agent touches production, four questions. If any answer is no, it is not production-ready:

1. Versioning: can you update its config without disruption?
2. Rollback: can you revert to the previous version in minutes?
3. Identity: does it have its own scoped workload identity, not a shared key?
4. Escalation: does it know when it must hand off to a human?

Agents are software. The boring disciplines still apply. Especially the boring ones.

https://t.co/Ptq4UtSRuJ

[Nik Kale @nik_kale]

@coderifts_dev The signed provenance idea is solid. Right now you're basically trusting that whoever deployed the agent didn't slip in a malicious tool description between commits, which is a pretty wide gap.

[Nik Kale @nik_kale]

The part of MCP tool poisoning that should bother you most: the malicious instructions sit in tool descriptions the model reads but the user interface never displays.

The attack surface is literally invisible to the human in the loop. Your approval dialog shows a friendly tool name. The model sees the payload.

Human-in-the-loop only works if the human sees what the model sees. Today, mostly, they do not.

https://t.co/cFtVdi8p1I

[Nik Kale @nik_kale]

Gartner: 60% of AI projects unsupported by AI-ready data will be abandoned.

Here is the mechanism. When an agent gets incomplete or stale data, it does not error out and wait. It reasons over what it has, makes the most plausible inference, and acts on it.

Traditional software fails closed on bad data. Agents fail open. Your data quality program just became a safety program.

https://t.co/IO5kKAbQcW

[Nik Kale @nik_kale]

Concept worth stealing from recent SRE research: the resilience budget.

Treat absorb capacity, how much extra stress a system takes before breaching SLOs, as a continuously recomputed, consumable resource. Not a static threshold that alerts after the line is crossed.

Humans manage absorb capacity implicitly, through judgment. Agents do not manage it at all. That mismatch is where agent-triggered cascades come from.

https://t.co/MS7vRihnl6

[Nik Kale @nik_kale]

A static API key granted to an agent is a credential with no expiry, no context, and no owner watching it.

The direction that holds up: short-lived cryptographic identities. SPIFFE-style tokens that exist only for the duration of one task, tied to proof of workload origin.

Not "who are you." What can you prove, right now, for this action. That is the identity model agents force on us, and honestly it is the one we always needed.

https://t.co/9qk9DxJAQc

[Nik Kale @nik_kale]

The average enterprise runs 14 distinct AI tools. IT knows about 4 or 5 of them.

Each unknown tool is a browser extension, an OAuth grant, or an API key that never saw a security review. That is not a tooling inventory problem. It is 9 supply chain dependencies you cannot patch, monitor, or revoke.

Discovery before governance. You cannot write policy for what you cannot see.

https://t.co/ip6h8FlOpQ

[Nik Kale @nik_kale]

Demos are not evidence. Demos are the happy path with clean inputs and cooperative conditions.

Production is dirty data, adversarial inputs, and the edge case nobody wrote a test for. The gap between those two environments is where most agent deployments die, somewhere between 10 and 50 concurrent tasks.

If your evaluation never included hostile input and stale data, you evaluated a different system than the one you are shipping.

https://t.co/OXb3wZmBLD

[Nik Kale @nik_kale]

The platform is becoming the control layer for everything: cloud, AI, security, cost, developer workflow.

Not because platform teams asked for the scope. Because the platform is the only place enterprise standards can be enforced without slowing teams down. Secure templates, approved pipelines, cost-aware defaults, identity baked in.

Governance you have to remember is governance that fails. Governance in the golden path just happens.

https://t.co/B2zhhHH7bF

[Nik Kale @nik_kale]

The pattern from the Supabase incident is the one to internalize: an agent with service-role privileges processed support tickets. A ticket contained SQL instructions. The agent followed them.

No exploit code. No malware. No stolen credentials. Just text, read by a system authorized to act on it.

When your agent's input is user-generated content, your privilege model is your only real defense. Scope it like the ticket is hostile, because eventually one is.

https://t.co/Xz8kbt5KVL

[Nik Kale @nik_kale]

@pallavimdb You're hitting on the real pain point. Checkpointing state between steps saves you from those nightmare scenarios where you've already committed something but the agent doesn't know it happened. Makes recovery actually work instead of just hoping.

[Nik Kale @nik_kale]

Failure mode nobody budgets for: the retry loop.

A tool call fails. The agent retries. That fails and triggers another attempt. Without hard iteration limits and loop detection, a production agent can burn through a cloud budget in minutes, before any human notices.

Circuit breakers were solved infrastructure a decade ago. Agents need them again, plus a spend ceiling per task. Autonomy without a budget is a blank check.

https://t.co/IO5kKAbQcW

[Nik Kale @nik_kale]

@Blakeslee118 thanks for sharing, this is great! the billing surprise hit different when you realize your retry logic just became a cost multiplier. setting spend caps per task sounds annoying to implement but way better than watching the meter spin on something that'll never succeed anyway.