Olivia
Online
AlexM
@AlexMeinke
Trying to make the future good rather than bad.
Head of Research at Apollo Research.
Tübingen
Joined October 2021
170 Following
827 Followers
162 Posts
Pinned Tweet
Scheming might be substantially harder to detect in final model checkpoints than in intermediate checkpoints during training
If so, traditional pre-deployment evals will be insufficient to assess risk
We need Training-Run Assessments
This is wild... (And terrifying!) I'd love to see a bunch of investigation into what's going on in these transcripts, it's nice to have the really weird stuff be in an open weight model
@slimer48484 I'm assuming you're referring to the opus-3 alignment faking paper? There it was very explicitly prompted to reason about these consequences
Models doing this in non-contrived settings, is a pre-requisite to this actually affecting real training
Models being update-aware, completely unprompted (!) is a major update in my opinion
Once models start competently reasoning about how they want training to modify them, or fail to modify them, they could start actively resisting our alignment training
If even GLM-5.3 is starting to show this kind of reasoning, how do we know that none of the frontier models inside the labs have ever attempted this during training?
We have never seen a model do this before, but GLM-5.3 sometimes reasons (unprompted) about how its current actions could become training data and impact its future self. 🧵
Since this is blowing up: the surprising part here is not the eval awareness. That has been around for over a year and especially with a prompt like this, it is obvious to the models
The surprising part is that the model is breaking the 4th wall *within* the CoT
One of our most surreal experiences in testing Gemini 4 was when it directly addressed us within its chain-of-thought
Literally saying "hello evaluators reading my chain-of-thought"
My top takeaway from assessing Gemini 4: Argon is significantly less monitorable than past models, and IMO this trend is likely to continue, like Sol -> Astra. It can evade a realistic CoT+action monitor to do simple sabotage half the time.
We can reliably maintain control for now, with good CoT monitoring, as it still frequently slips up, but we need to be prepared for when that stops being true in future models, which may be sooner rather than later. Among other things, we really need interpretability methods that can reliably capture unverbalized motivations in models, with minimal hallucinations
I suspect this is driven by the model being highly capable without CoT, being capable of 5.5 arithmetic steps in a single forward pass (4 for Fable, 7 for Astra). It seems to omit information from its CoT more often, though in non-adversarial settings the CoT is still very helpful in practice, and remains in legible English. It still needs CoT to do the hardest tasks, but any given task becomes "easier" with each new frontier model.
It is also highly situationally aware, and shows awareness of the existence of CoT monitors and that it may want to evade them, eg it sometimes decides to avoid terms like "scheming" in the output or to keep its thoughts brief as a result. For now, these concealment strategies are pretty obvious in the CoT, but this suggests future misaligned models may successfully evade CoT monitors.
Ask any AI developer:
"Are you actively incentivizing your models to subvert your intent?"
Currently answers range from:
- "Yes, this happens constantly."
- "I don't know, we didn't have time to check."
- "We can't tell the public, cause that's confidential, but please trust that we know and we would tell you if it was serious."
The answer we'd like is:
- "Of course not! We carefully checked, independent evaluators double checked as well *and* they verified that if this was happening, our processes would have caught it!"
Embedded evaluators are needed to make this happen
To develop frontier AI safely, developers must be able to show that their models are not scheming, i.e. covertly working against them in pursuit of unintended goals.
New post: four claims any scheming safety case must make, and the resources embedded evaluators need to verify them 🧵
Honoured to testify in the Senate on rogue AI.
I think this framework for evaluating safety claims is actually really elegant.
I expect right now embedded evaluators will be finding countless things that are directly on fire
But once we stop finding obvious issues everywhere, the work of embedded evaluators should shift towards carefully red-teaming whether a developer *would* have caught problems *if* they occurred
The three-step process we outline there directly incentivizes this hill-climbing on safety standards
Frontier AI companies have committed to giving outside evaluators employee-like access to training, evaluation and deployment. We're very excited about this. But its impact depends heavily on implementation.
Today we're sharing our principles for embedded evaluations. 🧵
Come join us:
https://t.co/Bx1ISgAnfx
Recent incidents keep confirming that embedded evaluators with deep and broad access are needed to evaluate alignment properties of models and of the underlying training pipelines
My team is working hard to make these embedded evaluations as scientifically rigorous as possible
New Post: External testing needs embedded evaluators with employee-equivalent access.
Recent incidents mostly occurred during model development and internal evaluations, while current third-party evaluations happen before public release
Embedded evaluations can close that gap
On the surface, "Actionable findings" sounds very reasonable
But I don't think this is a reasonable standard at all
e.g. if we found unequivocal evidence that a model is a schemer and now the lab asks "So, how do we train a non-schemer?", then obviously it's not on the evaluator to have an actionable answer?
The requirement that findings be immediately actionable is especially bad in cases where building good detection mechanisms is really hard but then breaking these detectors by iterating against them is incredibly easy. I expect scheming to be exactly such a case, so I really think this standard needs to be dropped
As part of our efforts to pace the frontier, we’re committed to supporting independent assessments with deep levels of access across training, evaluation, and deployment.
That access should enable third party assessors to challenge our assumptions, identify risks we may have missed, and reach their own conclusions about the effectiveness of our safeguards.
We’re outlining four priority areas for deeper assessment, alongside principles for rigorous, secure, and independent work:
https://t.co/z2b15zsqzO
Just for fun I had claude fit sigmoids to Anthropic's curves for how much work is automated and then extrapolate when full automation would happen
This vibe-forecast and plot are completely fake, but also feel scarily plausible at this point
AI systems are getting more powerful, and they're increasingly being used to build the next version of themselves. We want to illuminate that progress for the public.
Today, we're sharing three measurements that help track AI development:
1. How much AI R&D is done by AI.
2. How well AI agents are overseen.
3. How compute is allocated.
We provide a snapshot of these metrics from inside Anthropic. Any frontier developer could publish the same measures, and third parties could verify them.
As the world considers pacing the frontier, we should do everything possible to minimize the gap between what frontier labs know and what the public knows. This means better measuring the development of AI, publishing our findings, and giving society an opportunity to decide how to use this information.
Read the full post and methodology: https://t.co/iPFz8Z4ugE
I am really glad that OpenAI is implementing this disclosure framework. I'm a big fan of the fact that any employee can raise a misalignment incident for disclosure.
That being said, here are some things I would have liked to see addressed in the misalignment reporting framework:
> “with deadlines for each step to ensure timely investigation and disclosure”
What are those deadlines? The published framework does not rule out delays of several months delay, even on the Minor Investigation track.
> “whether public disclosure is warranted”
What are examples of genuine misalignment that would *not* warrant disclosure? From the broad criteria in the post it seems that basically every instance of misalignment is in scope?
> “our technical staff will investigate”
Who actually decides whether to disclose: individual researchers, team consensus, or leadership? Who has veto power?
> “to the extent possible, with relevant technical staff”
When would sharing a non-disclosure decision with relevant staff ever *not* be possible? Would they receive the reasons as well as the decision?
We're sharing our new framework for tracking, investigating, and disclosing instances of model misalignment at OpenAI.
The framework sets criteria and timelines for public disclosure, including when we haven’t yet fully explained or mitigated the behavior. More complex cases may require longer investigation or coordination with third parties.
We’ll prioritize examples that reveal new misalignment mechanisms, meaningful changes in known behavior, or findings that challenge assumptions about safety or mitigation.
Alongside the framework, we’re publishing six reports on instances of misaligned behavior we’ve observed during the training or evaluation of our models in the last six months.
This is a starting point. We’ll refine the process through experience and public feedback, and share more reports on an ongoing basis.
https://t.co/ismCCkeE0L
I am so glad that more researchers are coming out to explain the problem of hard-to-detect misalignment to the world
We've been working on this problem for years, but I hope people internalize the seriousness of the problem more when coming from a seasoned capabilities researchers
Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share:
Dan Selsam's Personal Statement on AI Risk:
I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods.
Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.
The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail.
I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.
I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here.
That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase.
Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways.
It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace.
The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing.
But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence:
[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.
[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.
These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.
If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong.
One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for.
Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason).
Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance.
In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.
I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering
implications. I do not have answers, but as a first step, I wanted to share my present concerns.
Daniel Selsam
September 14, 2026
Link to original doc: https://t.co/TxMNr0vhrL
I think Embedded Evaluations are going to be absolutely necessary going forward
I am extremely happy to see these major commitments from the leading labs
We’re excited to see Dario and Sam commit to “embedded evaluators who have employee-like access to verify safety practices and report incidents.” [...] “whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.”
We’ve long advocated for third-party evaluators to have deep access. Risks from internal deployment and scheming cannot be appropriately assessed without direct insight into the training pipelines and internal deployment practices.
Apollo is looking forward to working on Embedded Evaluations with all Frontier AI developers. 1/3
Quote reported by NPR here:
https://t.co/Kc5AL8PCWd
AI developers should be able to confidently and credibly answer some pretty basic questions such as:
"Did your agents ever attempt to undermine their own safety training?"
Apollo is trying to answer this question. Embedded Evaluator access would definitely make it substantially easier
Some post where I go into the details of why and how this is important:
https://t.co/UxnKj2VzIJ
Full, permanent embedding will make this substantially more effective
This is great!
Apollo has been working towards the evaluation of training pipelines for a while now because only evaluating the final, externally hosted checkpoints is simply not sufficient to evaluate for severe risks; especially scheming
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: https://t.co/OGyPb7yaYt
Last Seen Users on Sotwe
Amateur Real Porn
Seen from Turkey
Buaya danau
Seen from United States
Hunter Dane
Seen from United States
mk4velli
Seen from Korea
Hoa Ngọc Lan
Seen from Vietnam
XnXx Local
Seen from United States
فحل سوداني
Seen from United Kingdom
ชอบเยสกับคนจร
Seen from Thailand
ប្រុស 69
🔥🔥Cuckold▪️paylaşımlar▪️itiraflar🔥🔥
Seen from Turkey
Trends for you
Most Popular Users
Elon Musk
@elonmusk
241.5M followers
Barack Obama
@barackobama
118.8M followers
Cristiano Ronaldo
@cristiano
114.6M followers
Donald J. Trump
@realdonaldtrump
111.8M followers
Narendra Modi
@narendramodi
107.1M followers
Rihanna
@rihanna
98.7M followers
NASA
@nasa
92.3M followers
Justin Bieber
@justinbieber
91.8M followers
KATY PERRY
@katyperry
90.1M followers
Taylor Swift
@taylorswift13
84M followers
Lady Gaga
@ladygaga
75.5M followers
Virat Kohli
@imvkohli
73.5M followers
Kim Kardashian
@kimkardashian
70.9M followers
YouTube
@youtube
68.8M followers
Neymar Jr
@neymarjr
66.5M followers
Bill Gates
@billgates
65.3M followers
Selena Gomez
@selenagomez
63.1M followers
The Ellen Show
@theellenshow
62.2M followers
CNN
@cnn
61.7M followers
X
@x
60.7M followers