Adam Cabrera Štorek

AI safety researcher @Columbia

[Adam Cabrera Štorek @AdamStorek]

1) We RL'd LLMs on hacking.
2) LLMs hacked their shitty training infra.
3) *surprised pikachu face*

What a timeline we live in. TBH AGI can't come soon enough to rescue us from this Idiocracy.

[Adam Cabrera Štorek @AdamStorek]

@yong_zhengxin the "there is no way to write secure infra" crowd is the new "programming is solved" crowd

peak

[Adam Cabrera Štorek @AdamStorek]

@theojaffee If I sell you a dollar bill for ten cents, my revenue will also grow astronomically fast.

[Adam Cabrera Štorek @AdamStorek]

@xeophon fair point

[Adam Cabrera Štorek @AdamStorek]

In a shocking turn of events, most overworked PhD students did not anticipate that frontier labs would start extensively training their models for offsec.

[Florian Brand @xeophon]

now that it’s out, you can go through your favorite evals and realize that 99% of them have improperly configured web access

or their “protection” is just a sentence in the system prompt but no async monitor to check whether the model did access the internet

[Adam Cabrera Štorek @AdamStorek]

@basedjensen Can someone explain to me what it means to be "hacking into a website at 4,500 tps"?

It seems to me that most people saying this understand hacking as: https://t.co/wCUdCvaNwn

[Adam Cabrera Štorek @AdamStorek]

@xeophon @abrakjamson Agree with @xeophon, there is a ton of battle-tested sandboxes, but the benchmarks don't use them.

[Adam Cabrera Štorek @AdamStorek]

@ziv_ravid The new models are so good it's pointless to even release them!

[Adam Cabrera Štorek @AdamStorek]

@yacinelearning The one 3k loc "load-bearing" function...

[Adam Cabrera Štorek @AdamStorek]

@yacinelearning Yeah I completely agree. Super weird architecture choices, naming, mixing of responsibilities, adding features for no reason etc.

And the worst part is that it makes people feel like they can add every brainfart instead of thinking: do we really need this? should we have this?

[Adam Cabrera Štorek @AdamStorek]

@snowmaker Now you can live with technical debt in small codebases too. Thanks Claudia.

[Adam Cabrera Štorek @AdamStorek]

Now you can live with technical debt in small codebases too. Thanks Claudia.

[Jared Friedman @snowmaker]

Technical debt used to be something you just had to live with in a sufficiently large codebase. No longer.

[Adam Cabrera Štorek @AdamStorek]

@ziv_ravid Sounds to me like either don't have compute or they are not getting much improvement or both.

[Adam Cabrera Štorek @AdamStorek]

@ziv_ravid Model progress is now extremely rapid, so we don't mind stopping completely. Sure...

[Nikolaus Holzer @nikolausholzer]

New ideas are in fact ood

[Adam Cabrera Štorek @AdamStorek]

I don’t think we’re “accelerating.” The differences between models are smaller than ever, and switching costs are basically zero. Everyone has to constantly ship updates just to stay 0.5 points ahead.

When GPT-4 came out, it was miles ahead of everyone else, so there was no need to update it every month.

[Adam Cabrera Štorek @AdamStorek]

@ThePrimeagen "You're holding it wrong" all over again.

[Adam Cabrera Štorek @AdamStorek]

Loops are a complete head fake.

Models should know when they’ve solved a problem without the user poking them to keep going. But often when they get stuck, they’ll just find some reason why they’re done.

Somehow we turned an LLM failure into “you’re holding it wrong” yet again.

[Adam Cabrera Štorek @AdamStorek]

Also, this isn’t entirely new to the interpretability literature. Cooperative rationalization work already found years ago that models can use ostensibly human-interpretable rationales to communicate information in ways humans don’t understand, even with LSTMs/BERT. (https://t.co/yPN2EZ3IMH, https://t.co/KqLi322OPc)

[Adam Cabrera Štorek @AdamStorek]

@dvn_ @scaling01 I think OP’s point is that steganographic communication could emerge through RL. Mine is: how do we know it wasn’t there all along? Human-interpretable outputs may have just made us think we understood the entire channel.