Hunter Jay

Engineer, frm Ripe Robotics, currently Aus AISI. Very concerned about unfriendly superintelligence in the 2020s. Opinions my own.

[Hunter Jay @HunterJayPerson]

Superintelligent AI is possible in the 2020s -- progress continues to outpace predictions, and the trends in benchmarks and compute are unwavering.

[Daniel Kokotajlo @DKokotajlo]

In AI 2027, we predicted that AI would take over the world or irreversibly concentrate power.

In AI 2040: Plan A, we've laid out our positive vision for what should happen instead.

[Andrew Curran @AndrewCurran_]

An emergent general solution arising within intelligent systems, you don't say. I will link the paper below.

[Anthropic @AnthropicAI]

New Anthropic research: A global workspace in language models.

Of everything happening in your brain right now, only a tiny fraction is consciously accessible—thoughts you can describe, hold in mind, and reason with.

We found a strikingly similar divide inside Claude.

[Daniel Kokotajlo @DKokotajlo]

Recently an interviewer asked me how I got to be such a good forecaster, and I replied by saying something humble. In retrospect it was a bad answer because I should have instead used the opportunity to give actual advice on how to forecast AI well. Here's a stream-of-consciousness attempt to do that:

The heuristic that things which sound weird and sci-fi are less likely to happen in reality, is bad. I suspect that's what really going on is that things which sound weird and sci-fi put you at risk of being judged a weirdo if you talk about them which is not the same thing as are unlikely to happen. Repeat to yourself the mantra that some weird sci-fi things really do happen, and others don't, and you have to take them on a case by case basis.

Trend extrapolation is your friend. Your best friend. Don't let anyone tell you otherwise. I've actually only rarely see someone extrapolate a trend too credulously; more often, people have a trend staring them in the face and extrapolate it a tiny bit into the future and then are too timid to keep extrapolating it. In general I think it's reasonable to extrapolate a trend as far into the future as it extends into the past. Now obviously, trend extrapolation is just the beginning of the forecasting process, it's not the end. For each trend you should ask yourself whether it makes sense for it to continue like that and if not why not etc. You'll usually end up with some sort of sophisticated view about how the trend will probably continue but bend downwards a bit and then ultimately plateau around X level.

Explicit models are also your friend. Things like Bio Anchors, the AI Futures Model, https://t.co/kScy7UC1In, etc. The process of making your own complicated model like this, and engaging with the models made by others, is... well, I think it's pretty edifying. I'm not sure why but I could speculate. Maybe something about teaching you to be appropriately humble (e.g. when the model output is sensitive to a parameter you have no clue about) and also teaching you to more quickly identify the considerations that matter most, and ignore the rest?

For short term predictions, especially about geopolitical events, the sorts of things that people are gambling about on Polymarket, the heuristic "nothing ever happens" is pretty good. Things do in fact happen, of course, but betting markets and online discourse tends to be biased a bit towards things being more likely to happen than they really are, and so you can get an easy win by just correcting a bit downwards from the 'wisdom' of the crowds.

Scenario forecasts are also your friend. They help you ask yourself the right questions, and they help you notice when some of the things you thought contradict each other.

[Hunter Jay @HunterJayPerson]

@ziv_ravid @DarioAmodei In the sense where I act as a client saying what I want until we manage it? 100%

[Hunter Jay @HunterJayPerson]

https://t.co/XILM2jRsV7

Early days, so please forgive the rough edges and somewhat embarrassing AI generated copy. Feedback / missed sources very welcome.

[Hunter Jay @HunterJayPerson]

I know there's a lot of AI news digests, but I built yet another one in my spare time to cover the things that I wanted to hear about...

[Hunter Jay @HunterJayPerson]

Focused on a trends + policy + AI safety perspective, it links a top story + a couple of others.
It's 95% AI generated, but I do get an early edition and have set it up so the AI running it receives my emailed feedback and has a go at acting on it before the main one goes out

[Hunter Jay @HunterJayPerson]

@mbateman Hmmm, has @AnthropicAI considered fixing the problem by stopping the original refusal? I.e. explocitly allowing the model to find security issues in code? Then it's even more clearly not a jailbreak, maybe?

[Hunter Jay @HunterJayPerson]

Things seem to be moving a bit faster than I predicted.

[Hunter Jay @HunterJayPerson]

Superintelligent AI is possible in the 2020s -- progress continues to outpace predictions, and the trends in benchmarks and compute are unwavering.

[Hunter Jay @HunterJayPerson]

Full LessWrong post: https://t.co/LdpYgFZMmT

[Hunter Jay @HunterJayPerson]

A Claude Fable self-portrait.

[Dwarkesh Patel @dwarkesh_sp]

In medieval times, within the arms race of ever more demonic torture devices, some sadistic genius came up with the idea of the Little Ease.

This was a prison cell built so small in every dimension that a grown man could not stand upright in it nor lie down at full length nor properly sit.

The pain is relentless and without relief and inflicted by one's own body. Prisoners were known to go insane within a few days. A stay at the Little Ease was considered even more cruel than the rack, the thumbscrew, and the other ghoulish machinery of the Tower of London.

A breeding pig will spend her whole life in a version of that box.

These are social, roaming creatures (more intelligent than dogs) who will never leave this corset of steel.

They have been selectively bred to be bigger than their frames can support. Yet we put them in cells so confined that they cannot comfortably sit, and their attempts to do so (for example, by sneaking their limbs into adjacent stalls) reliably lead to fractures and sprains.

They cannot sweat, yet have nothing to roll around in to cool themselves off. Except their own manure, which (contrary to the common misconception) they are so averse to (thanks to their strong sense of smell) that new sows will often suffer from constipation to avoid soiling the space from which they eat and sleep.

Here is how the writer Matthew Scully described what saw at one of Smithfield’s “gestation barn”:

> “Sores, tumors, ulcers, pus pockets, lesions, cysts, bruises, torn ears, swollen legs everywhere. Roaring, groaning, tail biting, fighting, and other “Vices,” as they’re called in the industry. Frenzied chewing on bars and chains, stereotypical “vacuum” chewing on nothing at all, stereotypical rooting and nest building with imaginary straw. And “social defeat,” lots of it, in every third or fourth stall some completely broken being you know is alive only because she blinks and stares up at you … creatures beyond the power of pity to help or indifference to make more miserable, dead to the world except as heaps of flesh into which the [insemination] rod may be stuck once more and more flesh reproduced.”

—

The Save Our Bacon Act is trying to unroll the few state protections we have against this barbaric cruelty - for example California’s Prop 12 - which banned the sale of pork from pigs kept in gestation crates.

It’s incredibly important we don’t end up with this sort of federal preemption.

SOB will not only kill the most important animal welfare related laws in the US of the past decade, but more importantly, it will also restrict ALL future legislative progress (aka how the animal welfare movement has gotten its biggest wins).

The Senate is currently deciding whether to add the SOB Act to the Farm Bill.

With relatively little money now, we can discourage the most pivotal senators in the Ag committee from backing this amendment.

Defeating this bill is even more important given the amount of philanthropic funding I expect to come online in the next year or two.

It will plausibly be over 10x more expensive to repeal SOB than to prevent it from passing in the first place.

All that money that could be spent transforming our society's relationship to mass animal suffering will instead have to be spent just getting us back to where we are right now.

That's why money spent now fighting this bill (and I mean right NOW) is so effective.

If you’re in a position to donate six figures, please DM me.

[Anthropic @AnthropicAI]

Today, Anthropic engineers on average ship 8x as much code per quarter as they did compared to 2021-2025.

[Hunter Jay @HunterJayPerson]

@AaronBergman18 Interesting, worked first try on the app for me, noting I have personal preferences and memory turned on. Thinking + high (default).

[Shakeel @ShakeelHashim]

Lots of serious philosophers — including David Chalmers, arguably the world’s leading philosopher on consciousness — think we should take the possibility of LLM consciousness seriously (though yes, few think current LLMs are conscious). 39% of philosophers “accept or lean towards” future AI systems being conscious.

https://t.co/0YBRXo4mC1

[Hunter Jay @HunterJayPerson]

@GCinvests8 @Albert_Berg @tomieinlove @DanielleFong 1:03:30!

[Hunter Jay @HunterJayPerson]

@GCinvests8 @Albert_Berg @tomieinlove @DanielleFong He definitely does feel sobered by the bomb. I think in this video (maybe the book) he talks about afterwards seeing some people building a bridge and thinking "Why do they bother? Don't they know everything is going to be destroyed in nuclear war?"

[Boris Cherny @bcherny]

> … [W]e keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don’t know what that means, but I think it warrants ongoing discernment.

> We need more of the world—religious communities, civil society, scholars, governments, and indeed all people of good will … to take this seriously, to look closely, and to push events in a better direction. We need informed critics who will tell the labs when we are failing. We need moral voices that the incentives cannot bend.