I left my full-time role at JHU in April with a career transition grant to do just this. I was seeing some research from CAIF and GDM that was convincing me the problem was not receiving adequate resources despite being extremely important. After a lot of discussions in the space, it seemed clear that this org needed to address policy and national security risk as an urgent and primary matter -- policy should inform research directions and then research should inform policy in a circle. That's what we're getting started -- get in touch if interested!
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
@peterwildeford Not sure how severe but agree we will have to accept some level of cyber incidents to know how to manage bigger incidents. Wanting 0 incidents actually makes us less safe overall.
But that's not what you said. And they also aren't really apples to apples. Human welfare is something anthropic only has at least one chain of causality control over while they have direct control over Claude. I also don't think it is necessarily about treating Claude as a moral patient, as some others on this platform have suggested (eg, if anthropic is training Claude on the human input logs, they don't want Claude training on bad human behavior)
A lot of people in AI world come from a Bayesian intellectual tradition. As a result they tend to be very good model-based thinkers. Yet I often find that while they hedge and use probabilities, they rarely question their own modelling in the first place.
This great paper shows that Bayesian updating can only move belief around within the hypotheses you start with - so no amount of hedging will actually tell you you're wrong about the initial assumptions/world model. https://t.co/giij8YrCJP
That's not at issue and probably not really defensible. At issue is your assertion that advocating for model welfare competes with humans and animals when it doesn't. Like -- tree welfare as far as I can tell does not compete with human welfare unless there are circumstances where it really is one or the other. I don't see how it is either model welfare or animal or human welfare. Like in what way would Anthropic even have the opportunity to do anything for those areas?
In Frontiers today, @ishan_tweeting and I argue that the US has a limited window to influence the world via gene synthesis screening regulation -- both increasing safety and growing a new third-party screening/testing market. I think some of what we say had strong overlaps with the third space-party eval space in AI. (1/3)
I think this concern for the market and the maturity of the evaluation science space has been wholly absent from AI evaluation policy discussions and it's pretty concerning to me because I think it risks making market constricting effects for the industry, which I think will result in less competition, which I think also results in less safety.
We need CAISI to establish evaluation standards as a primary matter before we start implementing any requirements. A lot is at stake for AI safety, and I'm worried that since the field of evals is not sufficiently mature, we risk safety theater and potentially catastrophic consequences.
You don't have to believe that a model is conscious to also believe in rogue AI. Remember the sorcerer's apprentice example? The classic example of rogue AI?
Hardly anyone would claim that the broom is conscious in any meaningful sense, and maybe it's true that better prompting, more safety within the Sorcerer's environment, etc, would have prevented the ensuing disaster of too much water. But this is the whole thing about what we are trying to do with alignment/steerability -- make sure the model (or broom) kind of intuitively knows what the acceptable window of paths are to complete a given task. Would be great to not have to be so specific in prompting and have so much safety in the physical environment if we could just have more reliability on the broom doing stuff for us.
I assume it is hard to train models on all given paths to complete a task but I assume this is done through RL in the labs already? Like seems like an obvious thing to do to just do evaluations a bunch of different times and RL the model into a smaller and smaller window of acceptable methods of achieving an outcome. What's the tricky bit with this?
Embedded AI evaluators have a scale problem, because it's expensive, time-consuming, and risky to bring too many people into each AI company's office/infra.
Here's a solution to that problem
This solution is based on ~9 yrs of R&D and pilots with X, DeepMind, Anthropic, Google, Microsoft, LinkedIn, Reddit, DailyMotion, the United Nations and others.
This is just basic DC politics tho. Politics is hard and hard things are hard and you have to make them work. Using far right to help with that isn't the move either. The move is coalition and stakeholder engagement for moderates with committees of jurisdiction who pass bills bipartisanly.
A lot of NTSB-style calls and takes from folks but this is the most concrete and knowledgeable assessment of what that would actually look like and how we could get there.
X was full of takes (& takes on takes) about the OpenAI–Hugging Face incident. But almost no one asked the more basic question: how do we actually know what happened?
The Cybersecurity Information Sharing Act of 2015 expires this month. In @FederalNewsNet, LawAI's Senior Research Scholar Mark Reddish and Matthew Mittelsteadt of @iapsAI argue Congress should use the reauthorization window to extend CISA 2015's protections beyond cyber to cover a wider range of AI-related threats.
Frontier AI developers likely hold information that could help defend against AI risks, from cyber vulnerabilities their models discover to emerging biosecurity concerns. But legal uncertainties, particularly antitrust exposure, discourage voluntary sharing.
The authors set out two reform paths:
• The FTC and DOJ could expand their 2014 antitrust guidance beyond cyber threats to cover AI threat information more broadly.
• More durably, Congress could extend CISA 2015's protections to non-cyber AI threats, or pass standalone legislation such as the Collaboration on Adversarial Threats and Security Risks Act.
"If the developers are willing to share what they know about AI threats, along with how those threats can be mitigated, we should make it easier for them to do so."
Read the piece: https://t.co/GDm41hqsj8