We backdoored a model in a day.
Much of the talk about malicious models remains abstract.
We wanted to make this tangible, so we built the simplest version of the attack.
A few takeaways
Backoored models or sleeper agents are a real threat vector.
We demonstrated a naive implementation.
A well-resourced, motivated adversary can plant something more evasive than this and keep it dormant until the trigger is met.
Neither open nor closed models provide fundamental guarantees of behavior.
Alignment matters, but it is still a best effort, and far from bulletproof (hello FelonyBench).
An open-weight model is a CSV file of weights.
You cannot inspect declared objectives, hidden policies, or learned behavior the way you inspect a binary.
As models get better at long-horizon tasks, they'll get more useful and autonomous.
To unlock that utility, we will grant agents broader permissions.
Instead of writing a detailed plan, we will simply declare desired outcomes and let the agent reason about how to get there.
Those same permissions are what make agentic security hard and asymmetric.
How do you dynamically define the right scope, without constraining the agent?
Defenders will reason deeply over permissions, allowlists, policies, and more. An attacker only needs one direct or daisy-chained path.
This is not doomerism, nor is it an argument for slowing down.
Agents are a new form of intelligence.
Mapping legacy security solutions and frameworks onto alien intelligence won't scale or work in the long term.
Agents will force security to be rebuilt from first principles.
1/ dYdx Launch Incentives - End of Trading Season 1 🥇 🥈 🥉
We are excited to announce the end of the first trading season of the @dydx Launch Incentives Program and the start of season 2. Below, we provide an update on the status of the program and what to expect next 🧵
1/ @synthetix Market Manipulation
We've received outreach about today's market manipulation. This incident is isolated to the $TRB market, resulting in a ~2m loss to $SNX stakers. Let’s give some background before diving into the attack.
1/ Chaos Labs is excited to introduce the dYdX Chain Analytics and Risk Monitoring Portal! This platform is a pivotal piece in our ongoing efforts to support the @dydx community in the launch of the dydx Chain and the Launch Incentives Program.
7/ Our comprehensive analysis of the RAMM design for @NexusMutual shows strong alignment with strategic goals, and community-voted params are validated for launch.
Read the full research report here:
https://t.co/KR6o7nbjyu
I am pro-Palestinian. Some might be surprised by this due to my recent advocacy on @X for Israel, but you shouldn’t be.
I am anti-terrorist, not anti-Palestinian. It is not inconsistent to be pro-Israel and pro-Palestinian. And my pro-Palestinian viewpoint is not a new one. My pro-Palestinian perspective began more than 30 years ago when I was introduced to the Palestinian community and their plight in the early 1990s.
I have invested millions in helping promote Palestinian economic development and peaceful coexistence. We would do a lot more if we could be confident that the funds would be used productively.
The crisis in Gaza is largely due to a failure of leadership. The Palestinians elected Hamas in 2006 after Israel withdrew and evicted 9,000 of its own citizens from their Gazan homes. Israel withdrew from Gaza for peace. It was a small scale test of a two-state solution.
Rather than building the Singapore of the Middle East over the last 18 years, Hamas diverted funding to build tunnels, rockets and munitions to wage terror and war in an effort to eliminate Israel and kill Jews.
Like the Israelis, the vast majority of Palestinians want peace. They want opportunities for employment so they can earn a living wage to support and educate their families so the next generation can build a better life.
They want peace, beauty, happiness, health and prosperity as we all do. All of that would have been possible with Gazan leadership which focused on economic development rather than terrorism. Israel and (most of) the world wanted the Gazan experiment to succeed.
Israel simply wanted peace. Israel built a fence and created checkpoints to protect its citizens from suicide bombers and other forms of terror. The Egyptians built a concrete wall at their border with Gaza for the same reasons. The need for a fence and checkpoints is made self-evident by the catastrophic impact on Israel when the fence was breached on Oct. 7th.
Hamas is in the business of terrorism. Hamas makes money with grift, corruption, and funding from Israel’s enemies who support Hamas to achieve their own anti-Israel objectives. Hamas and those that support it don’t care about the Palestinians. The Palestinians are simply a tool to implement their anti-Israel and/or anti-Jew objectives.
Like other businesses, Hamas has a corporate hierarchy where those at the top make thousands of times more than the ‘workers’ at the bottom. Hamas’ leaders have put aside hundreds of millions and even billions for themselves.
Hamas uses their ‘culture’ of terrorism, cash, and other incentives to motivate young, brainwashed —often from youth—, radicalised militants to implement death, torture and destruction. Their ‘success’ at terrorism attracts more funding, amplifies Israel’s response, and the cycle continues.
Hamas does not care about the Palestinian people. Hamas knew with certainty how Israel would respond to the torture, rape, beheading, and slaughter of Israeli women, children, seniors and infants on Oct. 7th.
Hamas’ plan was to hide out in their tunnels and headquarters built under major hospitals, limit evacuations so that Palestinian citizens are exposed to the inevitable Israeli military response, and then rally the world against Israel in a globally coordinated response as innocent civilians die.
Israel has no choice but to destroy Hamas. It cannot allow its survival, as Hamas’ existence remains an existential threat. If ISIS invaded our southern border, we would do the same. We would warn civilians to evacuate and then we would go in and destroy the terrorists. We wouldn’t cease fire until they were obliterated.
The whole situation is an incredible tragedy. While I have always hoped for a viable and peaceful two-state solution, the Gaza experiment has been an abject failure. Future efforts for statehood for the Palestinians must learn from this catastrophe.
As always, I welcome your input, critiques and rebuttal.
What did I get wrong?
📕 "RWA Report: The Deep Dive into 2023 Market" 📕
After rigorous research, we're thrilled to co-publish with @redstone_defi!
RWA Tokenization will drive the next crypto bull market, and understanding the potential and risks of these markets is critical.
Let's dive deep 🧵👇
📕 "RWA Report: The Deep Dive into 2023 Market" 📕
We're thrilled to co-publish with @chaoslabs the most comprehensive RWA market overview featuring 31 projects, which is the fruit of over 1 month of research 🧐
Here's a glimpse of what's inside🧵👇
https://t.co/vP2vzMRatS
🚨 WE'RE HIRING 🚨
The Chaos Labs team is growing fast ⚡
We're looking for the best and brightest minds to join the team, with a focus on Data Scientists!
Feel free to reach out if you want to learn more about the open roles 💭
🇫🇷Bonjour, we're in Paris!🇫🇷
Reach out if you're in Paris @EthCC and want to talk crypto risk management, mechanism design, and optimization.
@omeragoldberg will be there with a bunch of the Chaos crew!
1/ Another week, another launch! We're thrilled to announce the latest addition to our roster of offerings at @chaoslabs: the @AaveAave Asset Listing Portal. This platform is specifically engineered to facilitate and optimize the integration of new assets into the Aave protocol
1/ Today, @chaoslabs unveils the Benqi Parameter Recommendations Platform, a cutting-edge risk management tool we created as part of our ongoing partnership with @BenqiFinance, an @avalancheavax
native borrow and lend protocol.
Our collaboration with @Uniswapfnd continues 🥳
We are proud to have received a grant to continue our earlier research on TWAP vulnerabilities. We'll explore different attack vectors exposed by Ethereum's transition to PoS, including pricing & block manipulation.
1/6
Genesis Timeline
November 8: "No material net credit exposure"
November 9: We lost $7M
November 10: Okay, we have $175M locked in FTX
November 16: Sorry, no withdrawals or new loans
November 17: Okay, we need $1BN
November 21: We'll go bankrupt without the money
👍