jon cooper

[John Scott-Railton @jsrailton]

NEW: malware developers added nuclear & biological weapons text to to their spyware.

Goal? To trigger LLM safety refusals... so that their spyware wouldn't be analyzed by an AI security scanner.

Cleanest practical example I can think of for why over-indexing on first order safety alignment is risky.

When closed (and open) models ship with aggressive refusals, they will be sprinkled with second-order blindspots that attackers will discover...and exploit.

We are only in the earliest days of attackers leveraging these features, and it wouldn't surprise me if users systems that need to handle complex cybersecurity issues demand that models be less safety-blunted.

In the weeds: @SocketSecurity's post also shows why intention matters in how you design a malware analysis pipeline to avoid prompt manipulation.

H/T to colleagues that shared this with me https://t.co/f3Aj9TYxU4

[jon cooper @jdc]

@alliekmiller It seems to be pretty common to view having been a senior manager as a sign that one lacks agency and is not a builder. Which, to your point, is nonsense.

[Justin Drake @drakefjustin]

Today a crazy quantum story just got wilder.

On March 31, the Google Quantum AI team published a landmark result on Shor's algorithm for elliptic curve cryptography. Technically, the paper was a bombshell: a dramatic 10x improvement over the state-of-the-art. As a stunt and wakeup call to the blockchain space, those optimisations were illustrated on secp256k1, the elliptic curve underlying Bitcoin and Ethereum signatures.

But perhaps the most striking part of the paper was sociological, not technical. Instead of following standard academic process, the optimisations were kept secret, hidden behind a zero-knowledge (ZK) proof. Google's accompanying blog post mentions they "engaged with the U.S. government". The ZK proof demonstrates the existence of algorithmic improvements without leaking details. Academic censorship with ZK, a historic first!

As a co-author of the Google paper I witnessed some of the context surrounding this censorship. To be honest, multiple aspects of that context don't sit well with me. As much as I believe the general public ought to know more, I am limited in my ability to whistleblow. Though let me be clear about one thing: the Google team's professionalism has been absolutely exemplary, and they deserve nothing but praise.

Censorship has a way of backfiring. The Streisand effect, where an attempt to bury something only draws more attention to it, is exactly what's unfolding today. First, Google's key optimisation has been rediscovered by the French. And in a thrilling turn of events, a collaborative Shor-at-home challenge just launched. The initiative, available at ecdsa[.]fail, breached a new Shor world record in a matter of hours.

Let's start with the rediscovery. Just two months after Google's paper, French quantum expert André Schrottenloher cracks the main secret optimisation. His paper, titled "Optimized Point Addition Circuits for Elliptic Curve Discrete Logarithms", landed on the arXiv today. Big congrats to André, who beat several other nerdsnipped experts to it. In a blog post also published today, Craig Gidney, the world expert on Shor optimisations, revealed that he'd been sitting on this very optimisation for a whole year under censorship pressure.

Interestingly, André missed a handful of minor optimisations, both from Google's original publication and from improvements found since. It's plausible there's still plenty of juice left to squeeze out of Shor, and this is exactly what the ecdsa[.]fail challenge is about. The verifier program developed for the ZK proof does double duty, automatically filtering for valid submissions. Dozens of compounding small and micro improvements are rolling in. As of the time of writing there's an 8.4% improvement to Google's circuit, as measured by the product of logical qubit count and Toffoli gate count. Nice!

The nerdsnipping ran deeper than anyone expected. Over the last few weeks it became clear it extended well beyond André and other quantum experts. Behind the scenes, a small army of amateurs quietly got to work. Inspired by Karpathy-style autoresearch, they turned AI on Shor. Ironically, the verifier program for the ZK proof makes an ideal reward function for AIs. The barrier to entry for this modern style of research is refreshingly low, with several non-experts, even a teenager, finding nice optimisations. Get in touch if you'd like to join a Telegram group with fellow autoresearchers :)

Part 2: neutral atoms and qday

The story doesn't end with Google. On the same day Google went public, a stealthy startup called Oratomic published its own Shor paper in a coordinated release. It made a splash, ultimately becoming the most upvoted paper on scirate[.]com, a website ranking arXiv papers.

Oratomic's claim was wild. By building on Google's logical optimisations and applying custom physical optimisations for neutral atoms, they claimed just 10K physical qubits were sufficient to run Shor's algorithm on secp256k1. That number is mind-bogglingly low.

Knowing essentially nothing about neutral atoms when Oratomic's paper landed, I was intrigued and decided to learn more about the tech. I fell straight down the rabbit hole and spent a couple hundred hours on the topic. I got a little obsessed and watched every YouTube video I could find and spoke to a bunch of experts.

My conclusion? The tech is real, very real. Even Google recently decided to start a neutral atom lab, a notable pivot from their sole focus on superconducting qubits. If you care about qday, i.e. the day a quantum computer will break the first piece of cryptography in production, neutral atoms demand your attention. I shared some of my learnings on Shor and neutral atoms in a 30min talk at the ZKProof cryptography conference. You can find it on YouTube by searching "zkproof neutral atom".

Here's an interesting observation about this duo of breakthrough papers: neither Google nor Oratomic say a word about what their results mean for qday. No timelines. Zero. Nada. That is especially baffling given that the whole point of whitehat quantum cryptanalysis is to inform qday estimations and help the general public make good decisions.

So let me attempt to partially fill the silence, similarly to what Scott Aaronson did in his April 29 post. Given everything I know, including scary non-public information, I now put the odds of qday by 2032 at 50%. 10% by 2030.

Anecdotally, the US government has its own date: 2035. Originating at the NSA and later adopted by NIST, it's when branches of the US government will be disallowed from using quantum-vulnerable cryptography. In plain language: with hindsight, that date is a joke and should be discounted entirely. I don't see how NIST avoids being forced to pull it forward by years.

Part 3: post-quantum cryptography

There are good reasons to sound the alarm today, but please do not panic. Rushing carelessly towards immature post-quantum cryptography is a recipe for disaster. IMO a good target date for migration is 2029, roughly 3.5 years out. 2029 happens to be the date selected by Google, Cloudflare, and the Ethereum Foundation.

These days most of my time goes to safely migrating Ethereum towards post-quantum cryptography as part of the broader lean Ethereum effort. There's a lot to do. We need to rip out and replace BLS signatures at the consensus layer, KZG commitments at the data layer, and ECDSA signatures at the execution layer.

The plan to get there is compelling, and is based on hash-based cryptography. Within the Ethereum Foundation we've developed a Swiss army knife called leanVM (github[.]com/leanEthereum/leanVM) powered by the magic of hash-based SNARKs. Thanks to truly exceptional work by Emile, Thomas, and others, its performance is derisked. Regarding security, leanVM is a jewel, a minimal zkVM crafted for end-to-end formal verification and maximum security.

Want to help? There are two $1M initiatives. First, the Proximity Prize (proximityprize[.]org). Solve a long-standing mathematical conjecture in coding theory, improve hash-based SNARKs, and go home a millionaire. Second, the Poseidon Initiative (poseidon-initiative[.]info), offers $1M for breaking Poseidon, the SNARK-friendly hash function.

[jon cooper @jdc]

@ChShersh Agreed on the "boring but slightly different tasks to practice" concept. That's what I tried to do here: https://t.co/NXtA715doC

[jon cooper @jdc]

@garrytan @ycombinator What is this new world’s “measure twice, cut once”?

[jon cooper @jdc]

@strikerglows @MartinShkreli Three very different shops, only one of which has a staff of 'investors' in this sense, meaning discretionary portfolio managers.

[Nix 🕊 @startingfromnix]

oh to write about a city the way E.B White writes about New York

"There are roughly three New Yorks. There is, first, the New York of the man or woman who was bom here, who takes the city for granted and accepts its size and its turbulence as natural and inevitable. Second, there is the New York of the commuter — the city that is devoured by locusts each day. and spat out each night. Third, there is the New York of the person who was bom somewhere else and came to New York in quest of something. Of these three trembling cities the greatest is the last —

The city of final destination, the city that is a goal. It is this third city that accounts for New York’s highstrung disposition, its poetical deportment, its dedication to the arts, and its incomparable achievements. Commuters give the city its tidal resdessness; natives give it solidity and continuity; but the settlers give it passion."

[Aaron Levie @levie]

Both Anthropic and OpenAI have new initiatives to help enterprises deploy AI agents within their organizations. This is a trend that’s early but going to get very big fast.

As agents enter knowledge work beyond coding, there is very real work to upgrade IT systems, get agents the context they need, modernize the workflows to work with agents, figure out the human-agent relationship in the workflow, drive adoption and do change management, and much more.

While AI models have an incredible amount of capability packed into them, there’s no shortcut to getting that intelligence applied to a business process in a stable way. This is creating tons of opportunities across the market for new jobs and firms, and the labs are equally recognizing the criticality here.

[jon cooper @jdc]

There is not enough attention paid to the espionage opportunities available via open weight LLM models. An adversarial or rival nation can trivially add subtle, difficult to find behavior.

Much much more easily than this (brilliant) cyber campaign 20 years ago. Great writeup.

[Lukasz Olejnik @lukOlejnik]

A 2005 state-designed worm designed to corrupt physics simulations sat undetected on VirusTotal for nearly a decade. Fast16, intercepted executable files at the kernel level and silently rewrote floating-point calculations to make them produce slightly wrong answers. Targets: high-precision engineering suites used for structural analysis, crash simulations, and physical process modeling, including LS-DYNA, a tool cited in reports on Iran's nuclear weapons research. The sabotage vector relied on deployment of the driver across a network via worm, corrupting calculations on every machine, and eliminating the possibility of cross-checking results against a clean system. Stuxnet got the documentary. Fast16 got twenty years of nothing. https://t.co/3qfJMziXVd

[jon cooper @jdc]

“goblin-affine reward signal”

lol

https://t.co/bzEEMDZrGh

[jon cooper @jdc]

@thsottiaux

[jon cooper @jdc]

@thsottiaux

[jon cooper @jdc]

@amedhat_ @patrickc Really appreciated this, all of which was new to me! Thanks.

[Elastic Security Labs @elasticseclabs]

LLMs have gotten good enough at reverse engineering to recover source code from obfuscated binaries with real accuracy.

So we asked the obvious next question: how fast and cheap is it to use one to build obfuscation specifically designed to beat it?

We benchmarked Claude Opus 4.6 against the Tigress obfuscator across 20 targets first, to map its strengths and failure modes. 40% solve rate. Phase 3 multi-layer combos hit 0%, with cost explosions that killed the runs.

Then we ran a dev/test/refine loop to build 3 purpose-built obfuscation variants targeting the same crackme, iterating directly against the model's known weaknesses.

The finding: LLM-targeted obfuscation is fast and cheap to develop. Context windows, budget caps, and shortcut biases are all exploitable attack surfaces.

The arms race just shifted.

[Ethan Mollick @emollick]

I have been using GPT ImageGen-2 for the past weeks

I didn't think that better image-generators would be a big deal but it turns out that there is a quality threshold I didn't expect, where you can now get text, slides, academic papers

Look at what it does with my "otter test"!

[jon cooper @jdc]

@SMB_Attorney 7.95 billion seems like a rounding error. Is that right?

[Air Katakana @airkatakana]

if i gave you the formula for coca cola right now and told you to go sell it, 5 years down the line you would probably have 0% of the soft drink market share. with literally coke

and yet everyone expects ai-coded apps to be in widespread use after less than a year

[Karri Saarinen @karrisaarinen]

https://t.co/srw9NQMkH2

[Tanay Padhi @tanaypadhi]

how did Allbirds pivot to AI compute hardware before the shoe company literally called ASICS

[sarah guo @saranormous]

I believe AI will deliver enormous gains to the global consumer: better products, better services, better healthcare, and tools that make ordinary people more capable, even superhuman. The upside is so large, and the geopolitical stakes so real, that we should move decisively toward it, not choke it off.

But people do not experience technological change as an aggregate statistic. They experience it through their bills, their communities, and their jobs.

So the issue is not whether AI will create value. It will. The issue is whether the path to those gains asks particular communities and workers to absorb too much of the cost upfront.

The institutions building AI cannot externalize the local costs of scaling and call future abundance the answer. If datacenters place major new demands on power and land, they should invest enough to strengthen the grid, ease pressure on bills, expand the tax base, and create durable jobs. And if AI compresses some of the entry-level work people used to learn on, firms should help build new on-ramps and training pathways into the new work that growth is creating.

This is not an argument for slowing the buildout down. It is an argument that rapid technological progress has to be socially durable.