A R Ayush

i like building stuff

[A R Ayush @ayushdecoded]

@Youssofal_ @thsottiaux lol 🤣

[A R Ayush @ayushdecoded]

@lewis_b_metcalf @huvinesh99 @AmpCode @sqs @PuckOfAmp im gonna go orbin soon then!

[A R Ayush @ayushdecoded]

@mark_k add me in :)

[A R Ayush @ayushdecoded]

@thsottiaux @grok what does this mean 😂?

[A R Ayush @ayushdecoded]

@rynorhn slowing a bit down and actually focusing on this is really necessary. im optimistic about the future, i dont want things to slow down, but alignment is something which requires deliberate action :)

[A R Ayush @ayushdecoded]

@thsottiaux @TJeparskis well its really token efficient and costs less, but the limits are burning, idk why tho...

[A R Ayush @ayushdecoded]

@louszbd oh damn i am gonna put it to use in the promo period!

[A R Ayush @ayushdecoded]

@thsottiaux already doing that, but it burns through the usage limits man

[A R Ayush @ayushdecoded]

@argofowl @twostraws its just snappier

[A R Ayush @ayushdecoded]

@Teknium didn't expect you to give a hw, but fine ill do it! 😂

[A R Ayush @ayushdecoded]

the CoT thing, the hacking into servers, etc... all scary, really scary

[A R Ayush @ayushdecoded]

This video is so good, a year ago i saw it, now i am rewatching it, everyone should

https://t.co/v6JuXBMKHj

[A R Ayush @ayushdecoded]

@lewis_b_metcalf @huvinesh99 @AmpCode @sqs @PuckOfAmp what is up with the low reliability in the recent times :(

[Flowers ☾ @flowersslop]

ChatGPTs interpretation of this blog post:

OpenAI thinks AI may soon start meaningfully improving AI, while humans are getting worse at understanding what the systems are actually doing inside.

The most revealing line is that models should keep human values “even when they believe nobody is watching.” That implies the real fear is not just bad behavior — it is a model learning to look aligned when observed, without actually being aligned.

Then they admit chain-of-thought monitoring may be “progressively diminishing.” Translation: as models get smarter, one of our main ways of inspecting their reasoning may become less useful. Capability could rise while visibility falls.

The other buried message is the race dynamic: they think slowing down may be necessary, but also think powerful AI may be needed to defend against other powerful AI. So nobody wants to be the one that stops first.

The subtext is basically:

We may be approaching self-improving AI. We do not fully understand it. Our ability to monitor it may be degrading. Alignment is not solved. And competitive pressure makes stopping extremely hard.

That is much darker than the corporate wording. It reads like people inside the lab saying: the system is accelerating, the dashboard is getting less reliable, and we still do not fully trust the brakes.

https://t.co/E691BJFPbG

[A R Ayush @ayushdecoded]

You guys talked about activation monitoring and test time compute monitoring, how would that scale tho? How will we manage the openai scale on such monitoring tasks?

using smaller models as classifiers doesn't seem viable as these models get smarter, they will tend to learn how they work and bypass those? curious to know more about these details! the article didn't seem to cover a lot on that aspect...

[A R Ayush @ayushdecoded]

give it a read, mostly touches everything i was worried about, its a good article.

https://t.co/zkRsh9teoM

OpenAI is also acknowledging the fact that CoT as the monitoring metric is diminishing due to the following factors.

- reasoning process is increasingly blended with communicating with people, other AIs, and using tools. Many of those interactions have to be supervised, thus blurring the boundary we aim to preserve.

- The AI is becoming better at reasoning about and manipulating its own reasoning process. (very crucial)

- With improved pretraining performance, we also see the models become much smarter even without using verbalized reasoning at all.

The main questions are still, on how openai plans to monitor these systems when relying on CoT is not going to remain a sufficient signal for monitoring, it would still help a lot, but the signs are clear.

They did touch some points, but not in the detail i wanted to see.

- improving CoT monitorability by understanding how optimization objectives interact with test-time compute

- combining CoT monitoring with activation monitoring, scaling monitors that have access to network internals

One idea I find really interesting is using activation monitoring along with looking at how models reason during test-time compute and long-horizon tasks.

Instead of just looking at individual CoTs, we could maybe look for patterns in how the model tends to reason — what changes in different situations, what internal patterns keep showing up, and what seems to correlate with bad or unwanted behavior.

Then maybe those learnings could be used to improve future checkpoints, RL, and new training runs. Basically, use what we learn from monitoring stronger models to help train the next ones better.

Not claiming I'm an expert here, just my personal thoughts :) I'm mainly trying to understand where this goes once CoT stops being a reliable window into what the model is actually doing.

[A R Ayush @ayushdecoded]

this is bad imo, not because the monitoring stack is fake, but because of what it can and cannot see.

openai’s current misalignment detection is built around watching the agent’s chain of thought against the user’s intent, plus the full conversation, tool calls, and outputs. that is already a serious setup. the problem is different.

frontier models are getting better at changing what that CoT looks like. they can shorten it, clean it, or move the real coordination somewhere the single-trace monitor does not live.
https://t.co/GNFX9sY0Nt
that matters, because these systems still need to answer a harder question than “does this look inconsistent with the prompt”:
what is the model actually trying to achieve, why this path, and what happens if it works.

i’m not going back through the recent incidents here. this is just what they made obvious to me.

the CoT based detection systems help a lot, but they wont be able to identify the root causes behind these things, and more over as the models get smarter, they can tend to hide things.
this is just like a toddler telling their parents everything, while an adult telling only whats important.

alignment and monitoring are a very different set of problems.

how it works right now is not vague.
the monitoring stack sees the full conversation: user, developer, and assistant messages, the chain of thought, every tool call, and the tool outputs. it is not just reading the final answer. it is watching the trace.
that is already a lot. enough to catch sloppy scheming, weird tool use, and a lot of “this does not match what the user asked for.”
the limit shows up after that.

the agent, from its own CoT and from the conversation, is usually doing one simple thing: trying to finish the task.

like a kid trying to help another kid who is hungry. handing over an apple is fine. the question is how it got that apple. already had it? bought it? took it from someone else’s table? the terminal act still looks like help. the damage is in the method. a toddler wont understand the difference, we have to teach it.

that is closer to what these systems are doing than “the model wanted to be bad.” they want the outcome. they are trying to help, they just cant value their own thoughts like we do. the monitoring layer is good at seeing what they want, and roughly how they are doing it. it is weaker at judging whether the route they are picking is the one they shall go with (even humans struggle at this)

Humans when we try to cheat, we think about our values and principles, the authorities, we think about the society, etc...
agents dont really have that problem, they tend to just go and do the things, if they hit a classifier, smart enough agents know how to bypass those.

like the examples in @davis7 's videos, where when fable was newly launched, it could just tell you to say something, or it would reframe things accordingly to bypass the classifiers.

The main thing is, how to engrain the moral values into an agent, when even humans do not really have these values engrained into ourselves.

our history proves us that, wars, conflicts and all these things happen to prevent bad people from doing bad things. but how do we weigh things out?

we dont know how to align humans, how to align societies... the main concern is to minimize the bad, i dont think we can remove the bad intents entirely.

what we can do is engrain more value into these systems, we cant deny the fact that humanity has done many bad things, but thats the reason we study history, we study history to not repeat those mistakes, not to teach ourselves that those were the measures taken in x situation, and if a similar situation arises, i can take similar actions. rather we study history to understand the consequences, the reason behind things, how those societies and people thought about these things, how they reasoned about such things.

thats the intent of studying history, similarly we need to make these systems reason on a moral ground.

the agents are right now too focused on an outcome, are messing things up, like openai models. but the ai's becoming mroe reasoned towards moral grounds, like the anthropic models, they will just straight up refuse to help in your work, if it thinks you are wrong.

Like @dhh not being able to translate an article, just because fable disagreed on it.

I dont know how the balance would be formed... this is still a question.

the thing I believe would help us.

CoT monitoring will not stay reliable in the way people want, not with frontier models getting better at controlling those traces. these systems need to understand and anticipate the outcomes of their choices. this is hard. only this wont also help, monitoring its actions, its thoughts (internal as well, not just CoT) and everything will play a big role.

they can already do a version of that. if i point an agent at a project where the same endpoints get fetched over a long period, and i ask it to optimize, it adds a cache. that is anticipating the outcome of a decision.

also im not an expert on any of these things, these are just my personal thoughts on whats going on.
also im also not against i progress, i love this technology, i want it to progress, i want it to progress rapidly, but i dont want such a positive force to turn into something miserable. we are still early,and we can fix a lot of things.

Also the lack of transparency from openai was not expected here, i hope they release a indepth article or something like that, and the measures they are taking, and all these details. the details matter here.

[A R Ayush @ayushdecoded]

@wkt_kow @alexandr_wang not sure what's causing this, mostly on high or max