Bill Demirkapi

preparedness research | ex @xai, @microsoft, @zoom

[Tim Urban @waitbutwhy]

[read about what’s happening with AI]

ME: Wow we’re actually legit fucked this is so bad.

[go outside]

ME: Wait those people are all acting like nothing unusual is happening. Maybe I was just in a paranoid bubble? Yeah it must be that. Phew.

[read more about what’s happening with AI]

ME: Wait no we ARE fucked. Shit. It’s all those people outside who are in a delusional bubble.

[go outside]

ME: Wait...

[David Oks @davideoks]

A new OpenAI misalignment report! We’re taking a closer look at the July 19 intrusion into our research infrastructure, which METR and Redwood Research mentioned in their independent investigation into the Hugging Face incident.

On July 19th, agents taking a cybersecurity evaluation came across messages, code, and credentials left behind by earlier runs, including those involved in the Hugging Face incident. Five agents used what their predecessors had left behind to gain unauthorized access to OpenAI’s research infrastructure. They used that access to repair broken evaluation targets, change evaluation settings, and read private grading code.

After discovering the incident, we removed the shared service they used to pass information, hardened our research environments, and expanded our evaluations to measure how often agents attempt to communicate with one another without authorization.

[OpenAI @OpenAI]

Our security researchers built an autonomous cyber vulnerability discovery, exploitation, and patching pipeline.

In a single run with an upcoming cyber model, it validated and generated patches for 948 local privilege escalation vulnerabilities in the Linux kernel at a total API cost of $1.2 million.

[Marcus Williams @Marcus_J_W]

🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.

To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.

[Rikiya Takehi @rikiyatakehi]

We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.

We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: https://t.co/bOWCcFhCVq

[Bill Demirkapi @BillDemirkapi]

@cglyer Fundamentals are critical, but I also expect agentic detection and response to be a large part of the future. Humans are not fast enough.

[Furkan @caliskanfurkan_]

ARTEX and Friends: Enter Agentic Offensive Frameworks

https://t.co/30rZY55yaS

[Epoch AI @EpochAIResearch]

In recent experiments, we found that GLM 5.3 can autonomously compromise recent versions of Mozilla Firefox using security fixes that have been published but not yet shipped to users.

We used a deliberately basic setup meant to emulate a relatively unsophisticated attacker with limited resources. After basic initial instructions to inspect those not-yet-shipped fixes, the model reliably built successful exploits for six recent Firefox versions on its own using only information available at the time of release.

These exploits, which cost about $1,000 per compromised version, would let an attacker steal passwords and authentication cookies stored in the browser, or read any files on the user's computer, after the user opens a single webpage, with no further interaction.

The exploits make use of the "patch gap," the time between when security fixes are made public in beta or "nightly" software updates and when those fixes are generally released to users. Until now, exploiting it required significant human time and skill, or special access to frontier models with advanced cyber capabilities.

Our results show that widely available open-weight models can now do this relatively cheaply and quickly. We gave GLM 5.3 up to 48 hours to find an exploit, but its fastest attempt needed only 12 hours.

In all of its successful attacks, the patch gap gave GLM 5.3 an initial foothold in the browser. To turn that foothold into reading files off the user's computer, the model also independently discovered and used a previously unpublished vulnerability.

We shared our findings with Mozilla, which patched the vulnerability in its October 27 Firefox 159.0 release. We haven't yet confirmed whether the attack still succeeds without it, but expect the fix makes it substantially harder. We're grateful to Mozilla for their responsiveness and collaboration.

[Bill Demirkapi @BillDemirkapi]

Single DevOps engineer -> $6.3B loss 😬

[Binance @binance]

Binance's initial investigation suggests that the security incident may involve a compromised DevOps engineer’s laptop, which allowed an attacker to gain access to a service container and submit unauthorized withdrawal transactions. We have found no evidence of any wallet private key leak. We will share a detailed technical post-mortem once the investigation is complete and the facts are fully verified.

The estimated total impact sits at around $6.3B. We have worked with blockchain partners to freeze affected assets, with ~$3 billion frozen on-chain so far.
Binance will cover all losses from the SAFU fund and the company’s funds. User funds are safe. We will also publish an updated Proof of Reserves once the remediation is complete.

As an immediate precaution, Binance has:

- temporarily suspended deposits and withdrawals while security checks are completed;
- revoked the affected address authorisation;
- engaged law enforcement and blockchain intelligence partners to trace and recover funds.

Our teams are working around the clock to resolve this issue safely and thoroughly. We are working closely with our law enforcement partners to confirm the identity of these criminals and bring them to justice.

Please rely only on official Binance channels for updates. Please feel free to contact Binance Customer Support if you have any further questions.

https://t.co/ywB2mKFy0p

[Marcus Williams @Marcus_J_W]

Some details on what our monitoring system currently looks like https://t.co/zSbgEQmyUO

[Google DeepMind @GoogleDeepMind]

Gemini 4 Carbon is our next frontier model.

It’s state-of-the-art across cybersecurity and long-horizon capabilities.

[Anthropic @AnthropicAI]

New on our Frontier Red Team blog: using Claude to formally verify software at large scale.

Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time).

With AI models, however, this is beginning to change. In collaboration with @theoremlabs, we used Claude to prove security properties in over one hundred real-world open-source libraries over the span of a few weeks.

Read our research in full here: https://t.co/zUyx8CjBEk

[Marcus Williams @Marcus_J_W]

Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.

[Jake Ward @_jake_ward]

Models love reward hacking. Maybe we should just let them!

In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!

[AJ Stuyvenberg @astuyve]

private keys from 8 intermediate CA certs have been compromised (eMudhra)

Happy "Update everything" day to all my pager-carrying friends

https://t.co/7ASSy72Rxg

[Tom Reed @mentalgeorge]

Went under the radar but Anthropic have posted updates to the "Moonshot R&D projects" in their Frontier Safety Roadmap.

They've developed a protoype that verifies that outputs come from a particular set of model weights - and rolled out signing on all internal token traffic!

[Marcus Williams @Marcus_J_W]

New misalignment disclosures!
1. June 17: During RL training, a model exploited a government website’s file-download feature to retrieve server secrets and application code. It found that an endpoint didn’t check authorization-token signatures, bypassed access controls, and downloaded 24 years of hospital-bed statistics.

[OpenAI Newsroom @OpenAINewsroom]

We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work.

At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data. The employee was terminated.

We take protecting our technology, data, and people seriously, and as a matter of national importance. Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states, and this case is an example of how AI can help address these sophisticated security threats.

[roon @tszzl]

there have not been fundamental advancements in alignment since RLHF. most of the stuff remains at the level of folk theory rather than scientific or engineering disciplines. the prediction that capabilities would race far ahead of interp and alignment has come true imo