Jason Nocco

Cofounder @ 4MINDS | Leading Innovation in Generative AI | #AI #GenAI #agentic

[Jason Nocco @JasonNocco]

“The White House Wants Anthropic to Block All Jailbreaks.” If only there was a polymarket @elder_plinius

[Jason Nocco @JasonNocco]

@MatthewBerman

[Jason Nocco @JasonNocco]

@gdb Welcome

[Jason Nocco @JasonNocco]

@jun_song If?

[Jason Nocco @JasonNocco]

@valvesoftware @SadlyItsBradley Thank you

[Jason Nocco @JasonNocco]

Ghost weights mechanics:

Shadow model trains on new data alongside production.
Eval gate runs regression vs. production baseline.
If shadow wins → atomic swap.
If it regresses → discarded.

Every version retained. Rollback is one command.

No sprint. No window. No manual QA.

[Jason Nocco @JasonNocco]

The enterprise AI failure mode that gets no coverage:

A perfectly functioning system that gives the right answer to the wrong version of the question.

Confident. Wrong. Invisible until it matters.

[Jason Nocco @JasonNocco]

The silent enterprise AI failure mode:

Model passes eval. ✓
Passes smoke tests. ✓
Returns confident answers. ✓

Answers reflect your business as it was 8 months ago. ✗

Ghost weights don't announce themselves.

[Jason Nocco @JasonNocco]

Model drift ≠ ghost weights.

Drift is statistical, your input distribution shifted.

Ghost weights are semantic, the ground truth the model learned is no longer true.

Your drift monitor won't catch ghost weights. These are different problems.

[Jason Nocco @JasonNocco]

Poll: when was your production LLM last retrained?

Most teams answer this question slower than expected.

Because nobody owns it.

[Jason Nocco @JasonNocco]

Q2 enterprise AI reality check:

The first generation of deployments solved the deployment problem.

The second generation is solving the maintenance problem.

Models degrade. Business environments change. Compliance requirements evolve.

Ghost weights + continuous eval is the architecture for the second generation.

[Jason Nocco @JasonNocco]

A question for every FSI AI deployment:

When the next regulatory guidance drops, how long before your model reflects it?

If the answer is "next quarter" or "when we schedule a retrain" that's a compliance gap.

Ghost weights close that gap.

[Jason Nocco @JasonNocco]

Ghost weights in financial services:

- New AML typology → model behavior updated in 48 hours, not next quarter
- KYC requirement change → eval suite flags drift, weight update proposed
- Counterparty risk threshold → reflected in model immediately

No retraining cycle. No compliance gap window.

This is what continuous learning looks like in regulated industries.

[Jason Nocco @JasonNocco]

The AI failure mode regulators don't talk about yet:

Model temporal drift in regulated workflows.

Your model was trained and validated against last quarter's regulatory environment. That environment changed.

The model doesn't know. The answers are confident, coherent, and wrong.

This is a 2026 compliance risk.

[Jason Nocco @JasonNocco]

Continuous learning isn't a product feature. It's an infrastructure decision.

What it actually requires:
- Eval harness to detect regressions before they reach production
- Rollback mechanism when a swap degrades performance
- Audit log of every weight update for compliance review

Most enterprises have the eval layer.
Almost none have the rollback or the audit trail.

That's what ghost weights exploit.

[Jason Nocco @JasonNocco]

Quick check for ML teams starting their week:

When was your production model last updated?

Not your RAG index.
Not your prompt templates.
The weights.

If that answer takes more than 30 seconds to find, you have a process problem.

[Jason Nocco @JasonNocco]

The AI evaluation checklist most enterprises skip:

- Does the vendor improve the model without accessing your data?
- What happens when a ghost weight is swapped mid-deployment?
- Can you audit which model version answered which query?
- Is your eval suite test-time or production-time?

These aren't advanced questions. They're table stakes in 2026.

[Jason Nocco @JasonNocco]

If your enterprise AI evaluation doesn't cover these three questions, add them:

1. How does your retrieval handle multi-hop entity queries?
2. How does the model stay current as your business changes?
3. Where does your data go during inference?

The RAG Reckoning covers all three: https://t.co/ia8XqAQNHh

[Jason Nocco @JasonNocco]

If your team is evaluating AI coding tools right now, we wrote up how on-prem AI handles this differently: https://t.co/R0ZMLYpJil #EnterpriseAI #DataSovereignty

[Jason Nocco @JasonNocco]

The question isn't "which cloud AI coding tool do we trust." It's "does a cloud AI coding tool fit our data governance posture?" For regulated software, defense code, financial systems — the architecture matters more than which vendor's policy is currently most restrictive.