Suha

Disclaimer: The views expressed here are solely my own and do not reflect the views of my employer or any organization I am or was affiliated with

[Suha @suhackerr]

@gaiarajan @Wikimedia congratulations!

[ayman nadeem @aymannadeem]

i get the vision of a software factory where engineers review specifications instead of code, but i think correctness and understanding are two very different problems and an AI-generated spec is the wrong interface for solving the latter.

formal verification can prove that code satisfies a spec but it can't establish whether the spec captures what we intended, or whether we've specified the right things in the first place.

i wrote about this in “plan mode is dead.” replacing thousands of lines of generated code with thousands of lines of generated natural language specs doesn't solve the comprehension problem. it just moves it up an abstraction layer. my eyes glaze over reading ai-generated text!

cheaper implementation means the cost of misunderstanding what we're building becomes more consequential.

we shouldn't need to inspect every implementation detail, but we do need interfaces that make intent, assumptions, and the boundaries of what's actually been verified legible enough to exercise judgment.

[Charlie Marsh @charliermarsh]

We built tar-codec: a fast and secure tar parser for the Rust ecosystem.

It's roughly twice as fast as our previous tar implementation, but with much lower exposure to parser differentials and other security risks.

Here's the state machine btw.

https://t.co/OeCuCwYrvP

[Gautam Kamath @thegautamkamath]

There's a new meetup in NYC on Reliable AI! Organizers include @karen_ullrich, @marksibrahim.

The first is November 16, on synthetic data. I'll be giving one of the keynotes.

There's also lightning talks! Sign up and tell the NYC reliable ML community about your work.

[Max von Hippel @maxvonhippel]

I am working on a map of the secure program synthesis space, in collaboration with @sergeybratus for our NDIST keynote on Tuesday. Here's a sneak-peak. The words "RISC-V" and "Lean" are used here WLOG, and some simplifications are made to fit everything on one page. Did I miss anything?

1. Develop semantics for each language in an interactive theorem prover, and then use these semantics to prove that every program is correct. (@oath_tech has done some work along these lines. See in particular Sam Xifaras' project on Ruby https://t.co/mSt5F5vyvw)

2. Develop semantics for an architecture, say RISC-V, or perhaps for all possible architectures, and then use this to prove that every compiled program is correct. (@AnthropicAI made headlines recently with their announcement of a very large-scale project along these lines in collaboration with @theoremlabs https://t.co/BWiBXyrAzH)

3. Develop semantics for an architecture, say RISC-V, or perhaps for all possible architectures, and then use this to capture a formal model of each existing compiler or interpreter, and then iteratively (provably) refine each such model to an interpretable and easy-to-use formal semantics; then continue as in approach 1. (See foundational work by @kirancodes at BASIS AI https://t.co/cH8qA22IFs)

4. Make Lean, or some other such ITP + programming language, fast to run (comparable to Rust or C), and then completely rewrite all software in it, and prove the rewrites are correct. (This is I think, loosely speaking, the idea behind @VictorTaelin 's Bend.)

5. Make implementation-level verification tools, such as Verus, more capable and ergonomic, and then use them to prove all software correct. (This is what @fhuszar and @Craig_Hughes_ are doing at Reasonable, & I'm aware that BASIS and AWS are also working on this.)

6. Design a language in which unsafe states are illegal, and then rewrite everything in that language. Here, what kinds of safety you can express will be much more limited but the problem is, conversely, way more tractable. (This is what @danwallach and @DARPA did with TRACTOR, where the safety property is memory safety.)

[Tinfoil @TinfoilAI]

"Popular" no longer means "Secure". We're at a point where pointing a LLM at any complex software is certain to find some vulnerability, no matter how many people use it.

What's left? We at Tinfoil have invested our efforts into two areas: using a very small amount of code and then throwing the book of security practices at it, frontier model review, formal analysis (very soon), fuzzing, you name it.

Where did we start? Firmware: the first code to run in our secure enclaves, taking instructions from an untrusted host. We replaced ~1M lines of widely used C (OVMF) with ~300 lines of open-source assembly and zero dependencies.

[Dr Heidy Khlaaf (هايدي خلاف) @HeidyKhlaaf]

Excited to announce that I'll be keynoting at Black Hat Europe this year in London! I'll be talking about the distinction between AI Security and AI Safety, and the co-option and conflation of security and safety methodologies in AI that are leading us down a precarious path.

[Hillel @hillelogram]

At @antithesishq we sell a deterministic simulation testing platform, and it's always been an interesting question exactly how we explain what DST is to people and why it's useful.

Here's an idea I just had and want to try out:

Imagine we vibecoded ourselves a poker platform and have a thriving userbase. After a while we start getting complaints: it is sometimes possible to change your bet after seeing the next player or card dealt. You can fold, see the next card, and then retroactively switch to a call. Obviously, we want to fix this as soon as possible.

Based on the nature of the bug, we'd suspect this to involve an interplay between the player's actions, their relative latency with the server, maybe how fast the server itself is running or how it communicates with a database. We'd want to try things like "user clicks a bunch of random buttons", and then see if the outcome changes as we adjust the latencies and server runtime.

Now say we do actions X->Y->Z->W, adding 100ms latency during action W, and find something "weird". Not the bug, but something that feels like it's close to the bug. We want to see if that sequence finds the bug if we change it to 200ms latency or add a bit more server load.

But now we have a problem. In a normal test, we'd have to adjust our test parameters and run the test from scratch. But then there are so many sources of nondeterminism and inputs that we're going to be in a wildly different system state by the time we've done action Y. We can't compare "doing W with 100ms latency" to "doing W with 200ms latency" because our timeline has already diverged!

So, with all that out of the way, deterministic simulation testing. The gist of it is we run the whole system in a simulated computer universe that's "nondeterministic on the inside", but "deterministic on the outside". So we can reproduce the exact sequence of events, with the exact timings and orderings, that originally happened with X->Y->Z, and only then add 200ms latency to action W.

Or, more conveniently, we wouldn't even need to start over. Once we got the weird state, we could undo W and retry it with 200ms latency and see if that gets us somewhere else.

This is good for state space exploration, but it's especially valuable when we actually find the bug. Once we find an error case, we can do causality analysis. In X->Y->Z->W, is the Z actually necessary or was it incidental? We can explore X->Y->W and X->Y->Q->W and see if those also have the bug, or if Z was necessary.

And it's valuable once we implement a fix! We want to be sure that X->Y->Z->W(200ms) now behaves correctly. In a normal test, it'd be excruciatingly difficult to reproduce those exact circumstances. With DST, we can just replay everything and find a fix.

There's a lot more nuance to DST and how exactly our tooling finds "weird states", but I think this might be a decent high-level explanation!

[Suha @suhackerr]

@renebrandel @getcasco happy birthday!!!

[Jenny Qu @GuanniQu]

i’m realizing a lot of vulnerability research is matchmaking. i find two components that disagree about what the same bytes mean and introduce them to each other. unfortunately they have chemistry, fall in love, and eventually agree with each other

[Calvin Qi @calvincongelado]

AI lingo guide

[cindy Φ @endgamebarbie]

grand tour nerdfest on international security, network politics, sovereignty studies 👀🌹 vroom vroom

[Pablo Antonio @PabloAntonio]

I'm doing an esoteric-PDFs meet up in New York City on Oct 17th.

Printed your most esoteric PDFs and exchange them for other esoteric PDFs. Hit me up for details.

[Tinfoil @TinfoilAI]

The @mozthunderbird team from @mozilla built Thunderbolt: an open-source AI client that gives users control. We're really excited it comes with @TinfoilAI as a built-in option!

[Miguel @Kaweees1]

If you die in SF, they don’t bury you; your body is used as an RL environment

[vicki @vboykis]

trying to imagine explaining to myself 10 years ago that we made a Markov chain so long that the pope is mad about it

[Aaron Grattafiori @dyn___]

This was a great, fun conversation with two of the buddies on a whole host of security topics and I've already been enjoying the other @OffensiveAIcon episodes they've put out, so also check those out. Thanks for having me on @ryanaraine and @juanandres_gs !

[Niko @nikogrupen]

The return of wake-sleep.

Wake-sleep algorithms were introduced by Hinton et al. in 1995 as a means of training neural networks by alternating two phases: a wake phase that learns from real data, and a sleep phase that learns from examples the network dreams up offline.

We're starting to see them come back in a big way in agent-native products. As agents solve increasingly long-horizon, knowledge-intensive tasks, they'll need to learn from their own work, reviewing graded records of what they did offline and turning them into memory they carry into the next task.

Sharing an overview of wake-sleep for legal agents and initial experimental results on Harvey LAB.

[sankalp @dejavucoder]

your agent after it's 50th compaction

[Dimitris Papailiopoulos @DimitrisPapail]

I know this may sound insane, but this is the best time to be a Researcher.