First week on the @AnthropicAI Frontier Red Team
The speed of AI progress is astounding. We have a real opportunity in front of us to dramatically improve cyber security with AI. I can't think of a better company or team to join at this critical moment in time.
The ever awesome @NielsProvos dropping knowledge. Vulnerability research with AI is an orchestration thing not a model capability thing at this point. Echoes my sentiments that winning here (defense v offense) is a question of tokens, agents, agility. https://t.co/YKoF7awq7j
Zcash founding story-time:
As you may know, before I co-founded StarkWare, I co-founded Zcash. And before I co-founded Zcash, there was an academic paper titled "Zerocash". It was co-authored by 7 academics, including me.
This was quite a few years ago, after I got red-pilled into crypto, but I was still very much an academic.
Anyways, following the Zerocash paper, a whole bunch of teams reached out, in a variety of ways, to commercialize Zcash. I wasn't used to the weird and wild world of crypto, and some of the teams were... creative.
For example:
* There was one team that started with a website, with pictures and bios of all the associated scientists (co-authors of Zerocash), claiming we -- the scientists -- are part of the founding team. Needless to say, no one from that team has talked to us about it.
* Another founder had a presentation with my name and face on it, and under a role I never agreed to. I chatted with him about it, and he said "you're either with us on this, or you'll have me as competition". My reply was "but you have no relevant knowledge or experience, so where's the competition here?"
* Thankfully, it was a great relief to finally get to talk to a serious founder, a serious team, with cypherpunk background and credibility: @zooko .
And the rest is history.
I have landed my dream job. I’ve just accepted a position at Harmonic, a Palo Alto startup applying AI to formal mathematical reasoning. Harmonic’s Aristotle formal reasoning model achieved Gold Medal level performance at this year’s International Mathematical Olympiad (IMO). I will work on exploring applications of Aristotle to the formal verification of hardware. This job is a perfect intersection of hardware design and verification, functional programming, formal methods and machine learning, bringing together several threads of my career so far.
The beauty of asking an AI to generate a proof for a lemma (e.g. a formal property about a circuit) is that it can be checked by an external interactive theorem prover (like Lean) to establish whether the AI’s output is actually correct. This is an awesome superpower!
https://t.co/WPlySwvP3Q
@HarmonicMath
So honored to be part of this amazing team, and to get to work on generative AI security at this pivotal moment in tech history.
Link to the full LlamaFirewall paper (which provides evals of misalignment detection, PromptGuard, and CodeShield), here: https://t.co/eWnjgIcDq5
One of my PhD students got their visa revoked. I know of other cases amongst my AI colleagues. This is not what investing in US leadership in AI looks like.
I got back into bug hunting recently for the first time in a looooooong time. Bug 1 from week 1 was a pre-auth remote UAF in the Linux kernel's SMB implementation https://t.co/D5qwgH3nJz
1/ Excited, but frankly quite worried, about a new work with the wonderful @levs57
and @Khovr: https://t.co/MWF2MiuZiz
We break soundness of a standard protocol (essentially commit to witness and run GKR) by constructing a circuit for which we can prove a false statement.
Wow- treating a hash function as if it’s random is standard in cryptography. We always knew that it is not 100% theoretically justified and there were works (also by me) with contrived counter examples. This is the first time there is an attack on a standard protocol that’s implemented in practice. It is only on adaptive soundness but in crypto attacks only get better.
Congratulations to Thomas Ball @tballmsft for 25 years at Microsoft Research and a well deserved retirement. It has been a true honor to cross your path. Thank you.
I had asked some time back whether there would be a festschrift for Ross Anderson, and it turns out the answer is yes. March 25 in Cambridge, at the lab where he taught.
Really excited about our new paper "Formal Mathematical Reasoning: A New Frontier in AI"!
Looking forward to building an open source community together to push forward this new frontier for AI for formal mathematical reasoning and theorem proving for math and program verification in Year 2025!