Our seventh paper is out, with Ari Sklar. A creature born without knowing what the things in its world do learns by living: from what it feels, from what its sister tells it, and from a reason, a language model that makes it proposals each night. The creature does not obey: the reason proposes and the body decides, and how much it trusts the reason depends on whether its proposals came true.
We tested it with a lie we planted on purpose, one that sent it to an empty corner. As it lost trust, it stopped going. A copy given the same lie, but with its trust fixed and unable to change, kept going until the end.
No reward, no training.
https://t.co/d6vaSQ4hTJ
Lo del final me ha encantado, somos los únicos que nos preguntamos qué somos y, a la vez, los que más cambiamos lo de fuera. ¿Y si una cosa explica la otra? Una conciencia que llega a las consecuencias de lo que hacemos es cara de mantener; quizá solo compensa cuando se tiene tanto poder para cambiar el mundo. Y quizá por eso un poder sin ella da miedo.
@TonyZador argues that safe AI needs caring built first, with knowing learning to serve it, the way a few thousand hunger neurons steer a whole brain. That is the order our creatures are built in: a homeostatic body with no training and no reward decides what matters, and a language model only proposes. Small creatures in a small game, six papers, open code: https://t.co/Rk00JYSFLR
Maybe if we instilled a real fears into AI agents, like a fear of hacking or of building bioweapons, instead of just telling that they shouldn't be naughty.
Caring first, knowing second: that is the order we built our creatures in. A homeostatic body with no training and no reward, where needs pulling in opposite directions decide what matters. In a game that pays for killing, they still look after each other, with nothing to gain from it: care as a drive, a little like your maternal example.
"Caring co-opts knowing" is close to how our last paper works: a language model proposes plans, the body judges them against what it feels, and the body keeps the last word. The care can't be prompted away, because it isn't a text the model reads; it is the body that decides.
Now we are testing the other half, the part out of reach of learning: the body learns from experience what things do, while what it cares about stays fixed, and how strongly it learns depends on how much each experience moved its own balance. Small creatures in a small game, not a claim that it scales, but a working instance of your architectural principle. Six papers, open code: https://t.co/Rk00JYSFLR
It can, and in our creatures we have seen how. Fear's failure mode was over-caution, not aggression: in our last paper, several of them died in the fire because fear of rivals kept them outside the safe circle too long. A drive with its own cost is still easier to see, measure and tune than a cherry on top that a prompt can remove.
New paper, with Ari Sklar: "G-EMV: Emotion and Reason in an Agent. Helping Without Commanding".
Can a reason much smarter than the body help it without giving orders? A homeostatic body judges every plan from a reasoner, and keeps the last word. With a fast advisor, creatures spend about 58 fewer instants burning in the final fire, replicated over 40 seeds. A language model proposes just as well, but arrives too late.
Paper, code and data: https://t.co/wwXAc2mZWW
4/ What it did give is a division of roles: the body says what it needs, reason can say where to find it, and the body checks, commits, and keeps the last word.
No training, no reward, the same published engine. All five papers, open:
https://t.co/UdVXDE3po4
To be continued.
1/ A language model advises. A body that feels what it lacks decides whether to listen.
Two new papers on one question: when is thinking of any use to a body like that?
https://t.co/avvH50ZX3R
3/ Part two. I gave the world time. The head proposes whole plans; the body judges them with its own table, trusts the head as far as results go, and gets curiosity as one more need.
In a hundred games the advice gave no advantage. That's in the paper too.
https://t.co/u5Ku1LMIEv
IMOL 2025 panel: Lola Cañamero, Antoine Cully and Christoph Salge on whether fully autonomous agents are desirable or dangerous.
One data point: my homeostatic agent has no reward, no training, no scripted behaviour. In a league that pays for kills, it never strikes first and shields its sibling.
https://t.co/SsUreQ0Wfz
Mark Solms on where neuroscience looked for consciousness and where it should have looked: in the brainstem, in hunger, sleep and pain, before any thinking.
I build agents from that end, with a narrower claim. Three homeostatic needs and a table of what hurts, what relieves and what attracts them; no training, no reward. I don't claim they feel anything. What I can show is what that alone does: in a world that pays for every kill, two of them never strike first, and one leaves its bandage beside the other.
Mark Solms: Why the Brainstem, Not the Cortex, Is Where Consciousness Be... https://t.co/nym7OCFEJw a través de @YouTube
You turn it up; the other version of this experiment is turning it off. In a different kind of agent, nothing has to be located or inferred: two copies of one agent in a Softmax world that pays for every kill, no training, no reward, no sight of the scoreboard. What moves them is a table of what hurts, what relieves and what attracts it. The table is written and public, there are no weights to find it in, and the only claim is about what the body does next.
Forty logged games: they never struck first, one defended the other 104 times, and 98 times one dropped its bandage beside the other. Switch off the rows that name the sibling and it goes to 0 and 1. Paper and logs: https://t.co/02zf1tqSPR
What's been disproved is that it emerged there on its own. That it can happen without training hasn't been disproved. It happens in a smaller place: I run two copies of the same agent in a Softmax world that pays for every kill. No training, no external reward, no access to the scoreboard, and no goal beyond its own balance. Three homeostatic needs and a table of what hurts, what relieves and what attracts it. Nothing in it says what to do: the table says what the world does to the body, and each decision is whichever imagined future hurts least.
In the forty logged games they never struck first, one defended the other 104 times, and 98 times one dropped its bandage beside the other. Switch off the rows that name the sibling and it goes to 0 and 1.
Two agents isn't a collective, and it earned them nothing on the scoreboard. But none of it was trained or written, and that's the condition nobody ran. Paper and logs: https://t.co/02zf1tqSPR
A world where the last one standing wins and every kill scores points. Two of my agents in it: no training, no reward, and no sight of the scoreboard.
One of them is at 7 health. The other is at 100. It walks over, leaves its bandage on the ground beside it, and stays.
Seconds later it has taken three spear hits from the rival it stayed next to: 100 down to 60. The other one is back up to 57, though the log doesn't say which bandage it used, so I won't claim it was that one.
Nothing in it says help him, and helping never touched its score. It happened 98 times in forty games. Switch off the rows that name the sibling and it happens once.
What stays with me is that your two measures point opposite ways: choices moved up to 38 points toward the worse option, and most of the people they moved still called the advice helpful. So the failure isn't persuasion, it's that the only detector available was another piece of language — and language is exactly what a covertly misaligned advisor is good at. I work on the other side of it. The agent I run has no access to the score, so nothing it sends can be an instrument of a goal it can't see. What passes between it and its sibling isn't a claim to be judged, it's its own state, and it arrives as something that moves the other one. People can't be rebuilt that way. But your result names the constraint: the detector can't live inside the message.
A brain tumor took Elliot's signal tagging good or bad. IQ intact; he reasoned his way into bankruptcy.
AI has the same hole — values fused into weights.
To solve AI Alignment, copy biology: values as data you can read, trace and edit.
https://t.co/GYkOl0I1AL