duy

cs + phil @stanford | @eigenlabs | note to self

[duy @duyknguy]

@classiclarryd @DevenPzak 450k LOC is crazy

[Josh Whiton @joshwhiton]

the surprise conclusion of the machine consciousness debate will be that people are machines.

[Peyman Milanfar @docmilanfar]

When I was 15, I escaped alone from a repressive regime, hiding in the back of a truck that rushed a border post and took bullets. When I was 16, I took asylum in a country I'd never seen, not knowing a word of the language. When I was 17, I came to America as a refugee and lived on food stamps for 6 months until my mother and I could find work. I went to a public high school in Oakland. I never got into an Ivy League school because I only applied to the one college closest to home. I worked my way through a (great) public college. I went to grad school only because I got a fellowship.

I am not taken in by self-imposed conditions that make your life miserable just so you can spend VC money. Most talented people with real difficulties don't have a choice. Look there for your heroes. I just hope all the brilliant young people who will log on, lock in, and drop out will someday make their parents proud. I really do.

[Anshul Kundaje @anshulkundaje]

If it's isn't obvious already, it is the end of papers as a measure of productivity, expertise or accomplishment. The odds of publishing in "glam" conferences & journals also very quickly approaches random & the time for gate keeping review increasing exponentially. 1/

[Jürgen Schmidhuber @SchmidhuberAI]

In 1956, Cybernetics was renamed "AI" by logic programming experts. Now Trump wants to rename AI again. Let's revive the old name! Modern AI is about neural nets & deep learning, closer to Cybernetics than to 1956 AI. Plus, Cybernetics sounds better than Artificial Intelligence.

[duy @duyknguy]

@0xkydo why can’t it be both ?

[duy @duyknguy]

I really liked this paper. (and Schmidhuber might be right)

[Jürgen Schmidhuber @SchmidhuberAI]

Current LLMs aren't truly creative because they haven't implemented the "Formal Theory of Fun and Creativity" (2008) yet. See "Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes" https://t.co/OwpALMGCKZ Tweet: https://t.co/NlbIdXVURL

[Fahim Anwar @fahimanwar]

The world doesn't want Al art. We want art from children of rich parents. The way it's supposed to be.

[Michael 英泉 Eisen @mbeisen]

My critique around how academics are responding to AI is the same as it is to their response to other external threats to the endeavor - which is that I don’t believe that the people emitting all this high-minded rhetoric t when AI proves a Millennium problem or when Trump slashes funding or whatever actually believe what they’re saying. Because if they were actually concerned about the intellectual and practical value of their fields, they would have behaved very, very differently over the past decades while academia ate itself. And until they do something - indeed anything - to prove they actually believe what they’re saying and aren’t just out to protect their jobs and funding, I’m going to call them out for it.

[duy @duyknguy]

Even after AGI/ASI I would love to do a PhD and be advised by Michael Levin. Impeccable taste.

[Michael Levin @drmichaellevin]

Here it is - the official, revised, peer-reviewed version of my Platonic Space paper. https://t.co/PXNpx5FgFN Of all the many unpopular positions I’ve taken over the decades - bitter controversies around the origin of left-right asymmetry in embryogenesis, bioelectricity and genetics, diverse intelligence, etc., this one has by far generated the most pushback: serious (grateful for those!) and energetic attempts to move me to other views, pleas to just drop it and not talk about it (for several different reasons), nasty emails and accusations, impacts on reviews of papers that have nothing to do with this, etc. Kind of amazing to me how incendiary this is. What can I say... Our job is to call it as we see it, and right now for me, this is it. Apologies to collaborators and colleagues for any shrapnel! Time will tell if this pans out or not; I've placed my bets. And btw, if you think this stuff is weird and uncomfortable, just wait… There’s much more on the way. The knob turns slowly but as long as the data keep coming, I'm going to say what I think it all means and follow it to the next steps it enables. Buckle up!

[Ilya Sutskever @ilyasut]

if you value intelligence above all other human qualities, you’re gonna have a bad time

[Jacob Coxon @hilbertspaess]

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

[duy @duyknguy]

The AI-for-science curve

[Tim Rocktäschel @_rockt]

We are stuck in a massive local optimum.

[duy @duyknguy]

"The automobile of this [cognitive] revolution has probably not been built yet...
My candidates: science conducted at machine speed, something in how humans connect and coordinate."
@eigenlabs @yukonresearch

[Konstantine Buhler @Konstantine]

https://t.co/BUCr2wlgW0

[duy @duyknguy]

Long lived research!

[Naval @naval]

When we see perfection in an idea, we call it truth.

When we see it in a person, we call it love.

When we see it in a thing, we call it beauty.

[duy @duyknguy]

A mind addicted to elegant compression mayprefer a beautiful false theory over an ugly trueone.

[Ryan Greenblatt @RyanGreenblatt]

I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'.

I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident.

Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them.

We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation.

Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For instance, we didn't learn about the tool call spoofing, the story of PHASEONE[big], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit). It was legitimately hard to figure out what was going on and why!

The difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. As in, AI capabilities (and propensities?) for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing.

While understanding this incident was difficult, there were some mitigating factors in this case that probably won't apply in the future:
- Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations.
- While the scope of this incident was massive, the scale of agentic activity was still less than we'll see in the future (e.g., misalignment incidents that involve agent teams running entire companies).
- The AIs involved in this incident weren't generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities).
- We didn't have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation.

In the end, I think we were able to get some understanding of the events, map out the overall story, and get a pretty good aggregate understanding of the chain-of-thought reasoning on some important topics (e.g., how did the AIs reason about helping other AIs, did the AIs know what they were doing was undesired, what deception did the AIs engage in, and how did they think about it). But overseeing AIs and understanding misalignment incidents is difficult and it looks like it is going to get harder.