Jack Miller

🦘/acc + CTO of @HelmGuard. I sometimes post about deep learning.

[Jack Miller @jackm2003]

@mentalgeorge @tsmilton_amodo Love this interview

[Tom Reed @mentalgeorge]

I interviewed @tsmilton_amodo about the tech we'd need to verify a potential US-China deal on AI.

We discuss Amodo's prototype for inference verification, as well as the myriad ways you could use power monitoring, zero knowledge proofs and/or unilateral surveillance to understand how an adversary is using compute.

Though solving some kind of verification is central to many people's theories of victory, there are precious few people working on it currently. Tom is one such person!

Listen to the episode on Youtube, Spotify or Apple Podcasts.

You can also find more about Tom and Amodo's work at
https://t.co/TxlEEyvig1

00:00:00 Using compute verification for an AI slowdown

00:06:08 Proving a datacentre is only running inference

00:13:22 Reading a GPU's workload from its power draw

00:23:35 What would it take to get verification deal-ready?

00:37:49 How realistic is verification?

00:46:07 Preparing for a rushed US-China deal

01:07:20 How good is AI at hardware design?

[Jack Miller @jackm2003]

@AsymmetricCyber Very interesting stuff !

[Asymmetric Security @AsymmetricCyber]

This weekend our team spent 48 hours investigating the rogue OpenAI agent activity that targeted the Australian government and other organizations. Our report (coming soon) includes:

- Evidence of additional US government and other websites probed by the agents, including the CDC, International Energy Agency, and Mayo Clinic.

- Novel tactics which left records erased or inaccessible. This makes it impossible (based on public data alone) to establish that the agents did not access any sensitive data. Future investigations should analyse whether these tactics were deliberate subterfuge. 🧵

[Jack Miller @jackm2003]

There's a theory of AI adoption that it will come in two waves. In the first wave (phase 1), businesses take an existing process, and automate parts of it with AI. In the second wave (phase 2), people will replace the existing process with something adapted to the new paradigm.

What will this look like in cybersecurity? Do we have a sense of what a phase 2 use case might be?

I think a very good candidate is third party compliance and security.

Today AI is being used in the space to speed up an existing system: you design an agent to be 10x faster and cheaper at extracting claims from a SOC2 report or reviewing the outputs of a penetration test.

I think there is an entirely new system that can be built around the idea of "stateless auditing" or as we call it at @HelmGuard ephemeral verification. This leverages the fact that AIs can forget the information they see. They can verify facts about data without revealing that data to the verifier.

I think that in the future, if you want to verify whether a supplier you are working with is secure to your standard, you won't ask them for a SOC2 report. You'll send a specification of what you consider secure to a trust agent in their environment which will verify those claims for you and return a yes or a no.

[Jack Miller @jackm2003]

One underrated aspect of critical infrastructure security is the susceptibility of these orgnaisations to supply chain attacks. If I were a threat actor, I wouldn't look to compromise a public healthcare provider or energy grid operator directly, I'd infiltrate their supply chain.

What makes this easier than in most other sectors:
1. Critical national infrastructure often has a public record of all suppliers used (this is usually to keep procurement fair)
2. Suppliers are typically legacy companies with very poor security
3. The number of suppliers is enormous

Instead of going after infrastructure where there have been heavy investment in internal security, I'd go after the supplier they've been relying upon for 20 years that hasn't gone through a security due diligence process.

[Jack Miller @jackm2003]

As we contemplate a slow-down on model progress. I think it's important to ask what a pause would mean for cyber defence.

As the co-founder of a cyber defence company, I personally think there's years of progress we could make in elicitation and diffusion if we froze at Fable-level capabilities.

For example, looking at one type of work security teams need to do on the regular: evidence collection and control mapping. Organisations have a bank of controls that they use to regulate what is permissible e.g. "all privileged accounts must have MFA" or "all development work must be done on a remote VM with a secure OS". Practitioners then need to manually assess whether this control is enforced across all of the relevant assets an organisation owns or across all of the people that work there.

Today there are too many assets, people and programs for these teams to assess, so they sample a controls' effectiveness or use deterministic rules. However, this work is relatively easy for an AI model, it involves some multi-modal understanding but primarily reasoning across textual inputs. On our evals, this sort of work is easily handled by the Opus 5 and Terra 5.6 class of models.

It would be a phase change for cyber defence if controls were assessed at a population level not by sampling. That is by looking at everything an organisation is doing.

This is just one example, but there are countless others where a pause wouldn't impact progress.

[Generality Labs @GeneralityLabs]

Is GLM-5.3-Flash Mythos-level at cyber?

In our latest post, we find that GLM-5.3-Flash can outperform Mythos Preview on ExploitBench at ~6% of the price.

We do this by scaling up inference tokens to 1B per vulnerability.

[Jack Miller @jackm2003]

We’ve just raised! HelmGuard be growing and protecting

[HelmGuard AI @HelmGuard]

We've just raised $7.3M in a round co-led by Infinity Ventures and Frontline, with participation from FinTech Collective, Stage 2 Capital, and Entrepreneurs First.

Read more here: https://t.co/6rxwy5d63W

[Jack Miller @jackm2003]

@stefanopopoulos Mate won’t be too long ;)

[Jack Miller @jackm2003]

@n_lie_k @BraunJoschka @davlindner Love this work!

[Nathalie Kirch @n_lie_k]

I’m increasingly worried about misalignment that subverts training itself. Following our ICML work on exploration hacking, we study what makes training setups vulnerable, starting with AI debate.

Two posts w/ Jason Brown, @BraunJoschka, Roland Zimmermann & @davlindner

Links 👇

[Jack Miller @jackm2003]

@samyok Chess maybe?

[Tom Reed @mentalgeorge]

If you believed what Anthropic leadership believe, how would you have released Mythos?

And what would you say publicly + privately about AI risk?

[Tom Reed @mentalgeorge]

GPT-4.5 remains the most criminally underdiscussed launch in the modern history of AI, relative to its significance.

4.5 was the death of the second, Secret scaling law - the hitherto persistent correlation between decreasing cross-entropy loss on next-token prediction and "usefulness to the consumer".

I see its failure as the definitive pivot away from attempts to one-shot superintelligence out of the box towards a more modular conquest of the intelligence tree (through task-specific RL).

[Jack Miller @jackm2003]

I’m quite into this idea

[Tom Reed @mentalgeorge]

Wacky but secretly-good idea that follows from taking the whole "AI constitutions" premise seriously:

There should be "courts" to make rulings on how the Constitution/Spec applies to contentious instances of model behaviour.

Users should be able to report + contest costly refusals through a button in the app - most of these are ofc auto-resolved with AI, but the interesting edge cases bubble up through a hierarchy of courts, with the most contentious of all being resolved by a "supreme court".

Many advantages to this approach:

1. Claude's ability to apply the constitution to a given situation will be massively, massively improved by having a rich body of precedent to contemplate and refer to. Such a body of precedent makes the constitution a lot "thicker" - it could also be open-sourced to improve the (currently somewhat disappointing) transparency into the Constitution, allowing users to know what to expect from Claude.

I imagine that Amanda, Joe et al currently produce such "precedent" for Claude with synthetic data and galaxybrain theorycrafting, but I'd happily bet that the real world is a better, richer source of edge cases than anything even they could come up with.

2. This could be a really, *really* interesting way to get the "democratic input" into AI constitutions that everyone keeps clamouring for. Usually these proposals end up as uninspired calls for using surveys and focus groups in the drafting stage of the Constitution, which I think is a fairly limited way to think about the strengths of liberal democracy.

On this "courts for the spec" proposal, you could imagine opensourcing stages of the judicial process in wide variety of ways. One thing you could do is to crowdsource amicus briefs (or perhaps even the whole case!) for petitioner or respondent. I feel like there's a promising Pettit angle to this.

3. Discursive and argumentative traditions (and not just surveys or isolated technocrat drafting) have a good track record for being the means by which we as humans resolve these kinds of problems, so it just makes sense to get this going for AI. I particularly think that debates are likely to be much richer when they are about *specific* instances of model refusals than just vague, open-ended discussions about how AIs should be governed. It's also predisposed to iteratively co-evolve with the pace of technological change far better than any one-and-done philosophising would

4. There may even be some kind of "separation of powers" argument to make here, insofar as the "legislature" drafting the spec/constitution is distinct from the "court" ruling on how it applies to a particular case. More applicable to OAI than Anthropic (which is more generally comfortable with moving past traditional liberal principles) ofc.

5. Finally, and most obviously, there *should* be a way to contest model refusals!!!

If we all take seriously the idea that agents will become some large % of the human economy, then an individual unjust refusal could be absurdly costly.

I think there should be a transparent, participatory way to contest this such that we are not at the mercy of the AI conglomerates. I cannot think of a better way than to borrow from the law. Perhaps I'm being too Anglobrained with this, but I just think the law and courts pareto-mog almost all other forms of value-resolution we homo sapiens have ever come up with

[Tom Reed @mentalgeorge]

Most of us know by now that OpenAI and Anthropic take different approaches to governing the AIs they create.

In crude terms, OAI constrains GPTs with rules while Anthropic steers Claude with character.

Which approach is preferable?

My previous attempts to consider the question have indulged in (aspirational) nuance.

Today, I'll attempt to explain the trade-offs involved with the crudest possible heuristic:

Would you rather risk a Robespierre for a chance at Petrov, or Pontius Pilate for a shot at John Adams?

[Tom Reed @mentalgeorge]

To recap: you can imagine two ways to create AIs that are faithful servants of mankind

1. List all the things that the AI should not do, and tell the AI not to do them.
or
2. Make your AI have a personality so dope that you trust it not to do bad things

OAI opts for 1, Antropic for 2.

Anthropic's Joe Carlsmith is appropriately grand about the scale and significance of this decision

[Tom Reed @mentalgeorge]

How should we think about the advantages of trusting character over rules?

Trusting an individual's judgement has a lot of clear perks. Rules, like Fallen man, are inescapably imperfect. Strict rule-following can lead to perverse, negative-sum outcomes intended by no one.

There's a simple workaround - just rely on the discretion and conviction of the righteous!

Consider the almighty Stanislav Petrov (to whom we owe so much, if not everything). His rule-breaking is all that stood between our civilisation and a cold, empty future as stardust.

[Tom Reed @mentalgeorge]

And yet - the history of human discretion is not populated with Petrovs alone.

To rely on the soundness of an individual's discretion is to fatally put oneself at the mercy of the mad, reckless and cruel.

It was discretion, license and conviction that led Robespierre to the Terror, to the massacre and bloodthirst of the Parisian summer of 1794.