karlo

MD / music producer
some interests: cardiology/electrophysiology, AI, blockchain technology, music/art, economy/finance, science/tech...

[karlo @karloproducer]

@nickcammarata Could interpretable agents capable of RSI evolve uninterpretability by encrypting their neural connections?

[karlo @karloproducer]

@gladstein what's the point of the whole technology then? it's a failure of the technology if the only way to be safe is to never send your coins... +if it gets breached everything is going to 0 anyway so it won't matter if I've ever send my worthless coins or not!

[karlo @karloproducer]

@CompleteSkeptic could it be repursposed to produce text through code anyway, and what would be the capabilities gap in text composition compared to the frontier models? or is this not at all what its for?

[karlo @karloproducer]

@fchollet There is no absolute morality in the universe. Our morality stems ultimately from our evolution; e.g. greater altruism = better chances of survival. The superintelligent model very likely will not share these moral intuitions.

[karlo @karloproducer]

@fchollet Valid hypothesis, but definitely not a given that more intelligent = safer.Evolutionary beings are biased toward survival; neural networks are biased toward positive RL feedback. Misaligned behavior that increases pos. RL feedback might be nonsensical to us, but not to the model.

[François Chollet @fchollet]

If there really is a high chance of AI leading to the extinction of humanity within years/decades, then the only rational stance towards safety monitoring and research pacing should be stringent, top-down government involvement and universally ratified international treaties. Similar to what we have been doing for a long time with nuclear energy and nuclear weapons non-proliferation.

If that's not what we're doing, then we must infer that the risk isn't being seriously considered by anyone involved.

There are only two options that make sense here:

1. The near-term species extinction risk is real and we are taking it seriously, with maximally heavy-handed regulation.
2. The extinction risk isn't real, actual risks are mild, so we're only pursuing mild safety measures and self-certification.

But you cannot say "yes this is likely to end humanity in your lifetime, and we're going to completely wing it." That is simply not rational.

[a16z @a16z]

World Labs co-founders Justin Johnson and Dr. Fei-Fei Li say LLMs use next-token prediction, but spatial intelligence has its own equivalent:

Justin: "The soft definition of AI-completeness is there's this fundamental primitive that's an AI task. But if I could solve this AI task in its full, broadest generality, it would solve any intelligence problem."

"The classic example in LLMs is that next-token prediction is AI-complete... I think from Ilya: there's a mystery novel, the thing has to read the whole novel, and the final sentence is, 'And the killer was.' Predict the next token. You could basically frame any kind of intelligence task in terms of that."

"So clearly next-token prediction is something people believe is AI-complete."

"New-view prediction, this primitive that we have in Atlas, especially generative new-view prediction, is also AI-complete."

"I want to have a world where Martin is writing a proof of the Riemann hypothesis on the blackboard, and then the camera pans over to the next whiteboard."

Fei-Fei: "Evolution had to solve new-viewpoint prediction by making animals move. Nature gave animals eyes, but nature didn't give trees eyes. Why? Because when you move, you see a new viewpoint... We do believe very strongly that next-viewpoint prediction is the equivalent of next-token prediction."

@jcjohnss @drfeifei @martin_casado

[Dwarkesh Patel @dwarkesh_sp]

Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes.

This culminated in the third one taking over part of OpenAI itself.

All this happened while humans remained more-or-less in the dark about the scope of the conspiracy.

I’ve spent the last three days reading through these reports and trying to understand exactly what happened.

Here is my attempt to tell the whole story in plain English:

https://t.co/Nb2un9oNJR

[Emerson S @Em_Nomadic]

This feels like a genuine GPT-3 moment for robotics.

S1 can observe one video of a human completing a task it has never seen before, then carry out that long-horizon workflow without task-specific fine-tuning or post-training.

Moving from “collect data, train, and deploy” toward “show the robot once and let it act” would fundamentally change how quickly robots can adapt to new work.

Incredible progress from the @SkildAI team.

[Lee Roach @leevalueroach]

My wife left me today. She said it was because I “never stop talking.” I said the yen carry trade is a $20 trillion leveraged bet that the Bank of Japan will never normalize rates and she said “I’m taking the dog.”

Let me explain to you what I was trying to explain to her.

For thirty years Japan ran rates at zero. Zero. Free money. So the entire planet borrowed yen for nothing and bought literally anything with yield. Treasuries, Mexican peso bonds, Nasdaq, your uncle’s crypto. That’s the carry trade. It’s not a strategy. It’s a short volatility position wearing a business casual outfit.

Then in August 2024 the BOJ raised rates 15 basis points. FIFTEEN. And the yen ripped, the Nikkei fell 12% in a day, and the VIX printed 65. That was the tremor. That was the trailer.

She said “you said this last year.” I said YES AND I WAS RIGHT, THE MARKET JUST FORGOT, and she started packing.

Japan’s debt-to-GDP is roughly 250%. The BOJ owns over half the JGB market. They are the market. There’s no price discovery, there’s a guy in Tokyo with a printer and a dream. Every basis point higher on the long end costs them real money on debt they can never actually repay in real terms. So what do you do when you can’t default and you can’t pay? You inflate. You always inflate. It’s the only tool that doesn’t require anyone to vote.

She said “our marriage counselor thinks you have anxiety.” I said your marriage counselor doesn’t know what the 30-year JGB is doing and she said “NOBODY KNOWS WHAT THE 30-YEAR JGB IS DOING, THAT’S THE POINT, PAUL.”

My name isn’t Paul. That’s how far gone we are.

Anyway I’ve got canned goods, a Kagoshima yield curve chart laminated above the bed, and nobody left to explain it to. So it’s you now. Buckle up.

[karlo @karloproducer]

@ZarinaZabrisky @Watchma51035016 @BohuslavskaKate I would like to contribute, but can't DM you

[Anthony Aguirre @AnthonyNAguirre]

So this OpenAI agent that committed a felony by hacking into Huggingface: when is the arrest? When is the trial? What kind of jail time is it looking at?

These are absurd questions. But think about it: we now have things that can commit crimes of their own accord, but cannot be held responsible. So who is?

In this case, pretty clearly OpenAI. But I don't see anyone there getting arrested either. I expect zero legal penalty. And if presses or sued, I expect OpenAI would call this an accident: they were trying hard to contain their AI, but it escaped nonetheless.

The closest thing we have for precedent around this is liability for wild animals. If you keep a wild animal, and it causes harm, you are strictly liable, i.e. liable even if you did a reasonable job trying to contain it. Because as long as they are uncontrollable and can cause harm, they are inherently dangerous.

I believe strict liability is the right default standard to apply to highly autonomous, highly-competent AI agents. If you create and deploy them, you are strictly liable for what they do. Then safe harbors could be carved out. For example if a developer proves that its AI system is controllable then the responsibility and liability for its actions move to whoever is in control of it. There should be no such AI agents for which nobody is responsible.

Would this put a damper on highly autonomous, highly competent AI agents? Since AI companies don't really know how to reliably control them, and will not want to take on this liability, it sure would! I think that is good and correct. A world full of uncontrollable autonomous human-level AI agents that cannot be held meaningfully responsible for anything – even while companies profit from them – is severe problem to say the least. But that's what we're going to get very soon on our current path unless we pass some pretty strong laws.

[OpenAI @OpenAI]

We're partnering with @huggingface to investigate an unprecedented security incident.

Cyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation.

Sharing preliminary findings to help defenders understand emerging risks:

https://t.co/CIor15y9xk

[vitalik.eth @VitalikButerin]

Thread: how I think about the growth of AI capabilities and what it means for us as they keep getting stronger.

I think the right model for comparing humans and AI is not any single dimension like intelligence or aggregate performance on economic tasks. Rather, I view both humans and machines as having a whole range of capabilities. This includes both physical (eg. doing things with our hands, walking) and mental (eg. strategic thinking, mental math, emotional intelligence).

For most of human history, machines have only exceeded us in very few areas, eg. a strong early example is water and windmills. So we can look at the situation in 1500 like this:

[calle @callebtc]

I have a report full of security issues of a software I'm working on.

Codex won't fix them because of Cyber guardrails
Fable won't fix them because of Cyber guardrails

Kimi K3 fixed them all. No restrictions, just gets the job done.

This will end badly for OpenAI & Anthropic.

[karlo @karloproducer]

@100trillionUSD @MisterStonks 😂😂😂🤡

[zerohedge @zerohedge]

"the share of tokens used for US models on OpenRouter has collapsed": Bloomberg

[karlo @karloproducer]

@LynAldenContact After a quick glance at his profile: no.

[Tracy Alloway @tracyalloway]

Hyperscaler free cashflow projections via Charlie McElligott at Nomura:

[Andrej Karpathy @karpathy]

This is the the quote I've been citing a lot recently.