Nay

rat/EA/AI/politics/quant follower

[Cole Salvador @ColeSalvador31]

Foreign Anthropic employees were supposed to lose access to Mythos 5 while export controls were levied. But it’s widely believed that these employees could still access “Mythos 5.1”: an even better model only available to employees.

There’s an undisclosed-models loophole. Our new piece for @A1Policy shows how the government might not gain access or knowledge of new capabilities quickly enough. Existing frameworks do not cover models unless they are soon-to-be-deployed. But keeping the best models internal-only for months is standard practice in the industry. There are four problems with this:

1) Lack of visibility. Cyber capabilities at the level of Mythos are already “must-haves” for agencies like NSA. It’s insane to believe that companies could be allowed to keep Mythos++ capabilities secret from USG.

2) Theft risk. Our adversaries want to steal top AI models and secrets. If the best models are secret from USG, an adversary could steal that model and instantly gain cyber, bio, and other capabilities MORE advanced than the best available to USG.

3) Use by insider threats. Our prior research has shown that probably over half of top AI company employees are unvetted foreign nationals. They could misuse highly capable internal models or distribute their secrets to our adversaries.

4) They’re hard to define. Company-only models change every day as they are trained and updated. This makes it hard for our policy tools, like export controls and the testing framework, to cover them.

Ilya Sutskever’s company Safe Superintelligence, for example, is widely believed to have near-frontier capabilities but completely skirts policies intended to manage cyber proliferation because it doesn’t publicly deploy models.

We offer policy options to close the loophole. The key update we need is for reporting, export controls, and USG access to start not when models are planned for deployment, but when models cross the capability thresholds outlined in the AI-cyber EO’s classified benchmarking process.

[Séb Krier @sebkrier]

I think these kinds of analogies essentially make a category error. It's a mistake to treat an AI as some sort of persistent situated entity with goals as one would a different species. A lion is a product of Darwinian selection, an AI is not; people port all sorts of biological properties to models but rarely make good arguments for why they apply. (Hendrycks did but I did not find that paper persuasive)

Imo Drexler puts it very well in Reframing Superintelligence: "Emerging AI technologies do not fit a psychomorphic frame, and are radically unlike evolved intelligent systems, yet technical analysis of prospective AI systems has routinely adopted assumptions with recognizably biological characteristics. To understand prospects for AI applications and safety, we must consider not only psychomorphic and rational-agent models, but also a wide range of intelligent systems that present strongly contrasting characteristics."

This doesn't mean that agents can't be goal pursuing or very dangerous, but agency with AIs is an optional, engineered, and bounded property, not an innate drive. Analogies to chimps/humans etc are mostly rhetorical, not actually descriptive. See also: https://t.co/S8QkQTp4Le

[Jack | amatica health @JackHadfield14]

Having worked in biotech & research for 3 years with no formal bio degree (my degree is in aerospace engineering), I’d say I know 10x more about biology than I do aerospace.

Degrees are useful as a stepping stone, but I think weight is heavily over attributed to them.

[Gappy (Giuseppe Paleologo) @__paleologo]

We are all part of the New Baseline

https://t.co/o7nfaBDbbF

[prinz @deredleritt3r]

In the age of RSI, the claim that models will commoditize looks increasingly dubious. The gap between the frontier and the second tier is already huge (much larger than the benchmarks suggest), is clearly growing, and will continue to grow at an accelerating pace.

Many will ask: but what about the plethora of enterprise tasks that don't need a frontier model? What if a fast/cheap model really is good enough for most knowledge work? The answer: RSI implies that the frontier labs will capture the *entirety of the pareto frontier*. They'll be SOTA on intelligence, but also on speed, and - if competitive forces so dictate - also on cost.

Fully automated AI R&D also likely means that tomorrow's models will look nothing like the LLMs of today. Some of the gap will consist of novel architectures or techniques, which the second-tier labs will struggle to independently discover and timely implement.

All of the above doesn't hold if RSI doesn't work! But if you believe that RSI will work, then model commoditization is likely the wrong bet.

[Alex Imas @alexolegimas]

I think we need a similar picture for RSI. If the 2023 definition of RSI (bottom right) was "can the model help improve themselves"? The answer is "yes".

But now we are probably closer to the middle left, where we need much more precise conceptualizations of RSI to have conversations about it and make progress on the economic implications. Model improvement is driven by many nodes (pre-training, post-training, hardware), and each of those nodes is interacting with others in "loops".

You can get "local" RSI within a loop, but depending on the complementarities between the loops, there may be relatively little impact on overall acceleration. Alternatively, if there are large complementarities across the loops, then RSI in one loop may get you closer to "global" RSI very quickly.

So when people say "do you believe in RSI", this may have been enough of a differentiator in 2023, but not anymore. Now it depends on what precise type of RSI that you mean. Timelines, moats, the economics, etc vary tremendously depending on the type of RSI being discussed.

[Dan Hendrycks @hendrycks]

The automation rate of remote projects has increased ~4x in the past five months.

[Jack | amatica health @JackHadfield14]

I spent about a month in bed, blindfolded, wearing ear defenders, barely speaking, and unable to tolerate normal light or sound.

What happened to my brain during that time was unexpected and extreme.

This is what it was actually like.

[Daniel Kokotajlo @DKokotajlo]

Recently an interviewer asked me how I got to be such a good forecaster, and I replied by saying something humble. In retrospect it was a bad answer because I should have instead used the opportunity to give actual advice on how to forecast AI well. Here's a stream-of-consciousness attempt to do that:

The heuristic that things which sound weird and sci-fi are less likely to happen in reality, is bad. I suspect that's what really going on is that things which sound weird and sci-fi put you at risk of being judged a weirdo if you talk about them which is not the same thing as are unlikely to happen. Repeat to yourself the mantra that some weird sci-fi things really do happen, and others don't, and you have to take them on a case by case basis.

Trend extrapolation is your friend. Your best friend. Don't let anyone tell you otherwise. I've actually only rarely see someone extrapolate a trend too credulously; more often, people have a trend staring them in the face and extrapolate it a tiny bit into the future and then are too timid to keep extrapolating it. In general I think it's reasonable to extrapolate a trend as far into the future as it extends into the past. Now obviously, trend extrapolation is just the beginning of the forecasting process, it's not the end. For each trend you should ask yourself whether it makes sense for it to continue like that and if not why not etc. You'll usually end up with some sort of sophisticated view about how the trend will probably continue but bend downwards a bit and then ultimately plateau around X level.

Explicit models are also your friend. Things like Bio Anchors, the AI Futures Model, https://t.co/kScy7UC1In, etc. The process of making your own complicated model like this, and engaging with the models made by others, is... well, I think it's pretty edifying. I'm not sure why but I could speculate. Maybe something about teaching you to be appropriately humble (e.g. when the model output is sensitive to a parameter you have no clue about) and also teaching you to more quickly identify the considerations that matter most, and ignore the rest?

For short term predictions, especially about geopolitical events, the sorts of things that people are gambling about on Polymarket, the heuristic "nothing ever happens" is pretty good. Things do in fact happen, of course, but betting markets and online discourse tends to be biased a bit towards things being more likely to happen than they really are, and so you can get an easy win by just correcting a bit downwards from the 'wisdom' of the crowds.

Scenario forecasts are also your friend. They help you ask yourself the right questions, and they help you notice when some of the things you thought contradict each other.

[Richard Ngo @RichardMCNgo]

AGI company employees should explicitly ask “how much wealth and prestige would I need to be comfortable leaving to do something unconventional?”

Because the actual answer is usually either “a level I already have” or “always more than I have”, which should prompt reflection.

[Matthew Zirwas, MD @MattZirwas]

I was wrong about the Midjourney ultra-sound scanner.

Well, maybe not wrong, but at a minimum I missed something obvious because I was thinking like a doctor who's been practicing for 25 years.

And I didn't explain my point well.

First, where I was wrong:

All historical precendent that showed that widespread screening imaging is net neutral or harmful was imaging that was expensive, inconvenient, gated by physicians and couldn't practically be repeated frequently short term.

If the Midjourney ultrasound is high resolution, harmless, inexpensive and convenient, people can get an initial scan, then if there are abnormalities concerning for cancer, they can get weekly follow up scans to see if it's growing/changing, and if it's not, they can leave it alone.

In retrospect, that is obvious but it never occurred to me.

Now, you'd assume that that approach would have to lead to it being useful and saving lives, and it probably will. But we won't really know it does until we have a couple years of data. Lots of things that seem obvious in medicine end up being wrong once we collect data.

Second, what I didn't explain well:

It's not that I think non-doctors are 'too dumb' to use the results effectively.

Its that historically it was literally impossible to use the results effectively, and that is super, super counterintuitive. It seems obvious that finding stuff early is beneficial, but experience has shown that it isn't.

Here's why:

The vast majority of abnormalities (i.e. possible cancer) isn't cancer - like over 90% of them, ends up being harmless - something thay your body could have handled on it's own.

But the only way to find out was to have invasive, risky procedures to biopsy or remove what was found.

And overall, the side effects from all the risky, invasive procedures to track down the over 90% of stuff that was harmless equal or outweigh the benefit from removing the less than 10% of stuff that wasn't harmless.

If the MIdjourney device can be repeated frequently, like weekly, at a low cost and is harmless, it could negate the need for the risky, invasive procedures.

Not saying it will, but it seems like it could and I confidently posted yesterday that it was a bad idea.

I was wrong to confidently post that.

[Richard Ngo @RichardMCNgo]

To elaborate on what (I think) Michael is saying: if you lived in a deeply trustworthy civilization then when you observed a problem you could just go fix it directly.

But if your civilization is actually the thing getting in the way of you solving core life problems (like raising healthy children, solving ageing, building high-trust communities), then your options narrow to either:
1. getting into a conflict with established power structures (scary for scrupulous people!)
Or 2. finding some decisive source of power such that you can win without ever admitting (even to yourself) that you’re in a conflict.

On an emotional level, planning around RSI allows you to dream of future where you either win overwhelmingly or lose overwhelmingly. You never have to do the hard, risky part.

[OpenAI @OpenAI]

GPT-5.4 helped drive a medicinal chemistry project from literature review to a validated experimental result.

Paired with https://t.co/gcDaph8b2B’s Maria AI and specialized lab, the model proposed an unexpected way to improve a widely used reaction in drug discovery.

[Nay @DoomNayer]

@DrPhiltill Until we have Dyson Swarms. Although Earth's land will have historical value

[Jim Fan @DrJimFan]

Today, we enable AutoResearch in the physical world for the first time! Introducing ENPIRE: we give 8 Codex agents a fleet of robots, an allocation of GPUs, and generous token budget. We set them free with a simple goal: solve the task as quickly as possible, keep the robots busy but stay safe, don't waste precious compute. Make no mistake.

Then humans step aside and our watch begins. The robot fleet starts to come alive: they learn to look for visual clues, reset the scene, practice novel skills, tinker with control stack, read papers online, debate, reflect, get stuck, and try again directly on the hardware. All we did is to give Codex an API to the world of atoms, and the rest is emergence.

ENPIRE is able to solve high-precision tasks like tying zip-ties, organizing fine pins, and installing GPUs all by itself. We also discovered a new type of "physical scaling": 8 robots exploring in parallel improves significantly faster than fewer ones.

A part of our NVIDIA GEAR lab now self-improves tirelessly over night. We just read the reports in the morning.

/goal: we all take a holiday and Jensen wouldn't even notice ;)

We will be open-sourcing everything, so you can host your self-running robot lab at home too! Deep dive in the thread:

[Peter Wildeford🇺🇸🚀 @peterwildeford]

People really hate data centers...

Would you want a nuclear energy plant in your area? 53% oppose, 45% support

Would you want an AI data center in your area? 71% oppose, 27% support

Incredible

[Nay @DoomNayer]

@nickcammarata Does it make you significantly more productive though?

[Dean W. Ball @deanwball]

Precisely as I predicted, the recent cyber EO, which admin officials insisted was not a licensing regime, ends up in practice being a licensing regime. Forget “voluntary,” forget “permissionless.”

AI is licensed now, but the requirements change constantly and are always a secret, even to the administration itself, which will discover the rules spontaneously in real time as it reacts to events. This means also that the rules are in practice stricter and more roughly enforced for organizations the administration does not like.

Can you blame Anthropic for making itself so disliked? In a sense, sure. The problem is that this childish “he said, she said” is all we have to go on in our analysis of the situation. And because there is no transparency (it is all calls and texts between “White House officials” and “Anthropic executives”), in practice it comes down to who you trust more.

This is why we create laws! To abstract away from personal power struggles and grudges, to submit to the steady application of rules so that complex human activity can unfold with predictability.

The rule of law has been being eroded in the U.S. for my entire life, but it is especially acute in AI because of both the lack of much preexisting law to serve as bulwark, and because of this admin’s insistence that it is Not Regulating AI. This has become an excuse for vagueness and evasiveness in rule-drafting (see the cyber EO), and this in turn makes the lawlessness worse.

The government wants to apply its force to frontier AI, that much is clear. It wants to make the industry submit. And in service of that goal, it has discovered that “not regulating AI” is in fact a great excuse for refusing to support laws that could constrain the admin’s exercise of power. In other words, “not regulating AI” is a *justification* for the tyrannical control of AI by the state.

This should alarm you regardless of what party you are in. What you are seeing now will be used against you one day soon, if not by this admin then by its successors. This is the antithesis of the rule of law.

The administration cannot and will not fix this problem alone. We need Congress to step in and impose rules on this mess.

[Peter Godfrey-Smith @pgodfreysmith]

More on the "If Anyone Builds It, Everyone Dies" AI-doom book by @ESYudkowsky & @So8res. Finishing it on audio the other day, it was coincidentally followed by a radio story about computer security, Mythos, etc. Found myself thinking that the more likely malign path from here is breakdown of economic life, not world domination by a single rogue system.
I repeat, though, that it's a very good book.

[Richard Ngo @RichardMCNgo]

I suspect the Manhattan Project was a mistake even given uncertainty about whether Germany would build nukes. It’s just such an enormous escalation, and set the stage for a hugely risky and harrowing Cold War.

The “we’re in an AI race with China” frame now feels similar.