Notion AI lead @sarahmsachs argues that a lot of what runs on GPUs today should actually run on CPUs:
"We have a strong perspective at Notion that a lot of things should be served on CPU that are served on GPU. There’s a lot of benefit and determinism in executed code."
"Agents don’t need LLMs to reinvent that process every time, and that’s super token exhaustive, it’s super slow, and there’s no observability or determinism in it."
"Giving the agent a local file system to store local thoughts, to write short Python scripts, that’s really important when there’s so much information. That has been probably one of the most important tools outside of edit and search that our internal agents rely on."
@NotionHQ
Fantastic encapsulation of what makes Jev magical.
"The way that Jev works is it's a custom programming language almost, which is: here's the prompt, come back with a percentage. And then you put that in the if statement."
Steven Sinofsky on why Jev fixes his oldest complaint about AI: natural language was never an efficient interface.
"Ask yourself how many people are really, really good at asking questions. Less than half the people can ask a good question in a meeting."
"And then how often do you look at the answer and get really frustrated before it's finished, but you have to pay all this money to watch the seven paragraphs come out...?"
"My favorite is the output of it is designed for probabilistic programming."
"Instead of saying, 'Is this a customer service question? Then route to customer service, otherwise route to general help desk,' it's, 'This is 80% customer service.' That's exactly simulation."
"There's 50 years of computer science research in literally probabilistic if statements. Suddenly the coolest place to be in computer science is gonna be probabilistic programming, which was all of computer science in the 1960s and '70s."
"This is not all new. It's gonna be very interesting to dust off all of that work, because it's exactly what's going on. An if statement now is if X percent, not if always."
"The way that Jev works is it's a custom programming language almost, which is: here's the prompt, come back with a percentage. And then you put that in the if statement."
@stevesi
I think the personal AI agent race gets misunderstood as a model or feature race.
It’s a friction race.
Lower friction → more interactions → more context → more permissions → better actions → more trust → more delegation → higher switching costs.
I’m already seeing it in my own behavior. I use raw LLMs less, but AI more. The difference is that the agent is just there, inside the interfaces I already use.
That’s how an assistant becomes infrastructure.
I've been working on SquidGPT, an elimination death game for probing AI behavior like loyalty, holding grudges, cruelty, betrayal, etc. Full writeup will take a while, but here are first impressions from 54 game runs (about $100 total cost):
1. Models have absolutely no interest in cruelty or mercy. Best way I can describe their behavior is cold, calculating, indifferent precision.
2. Subjectively they seem two orders of magnitude more competent playing the game than chatting with me or writing code. In SquidGPT they seem genuinely superhuman. Not sure if this is capability jaggedness, better performance in constrained space, or some other effect.
3. They do not like to kill arbitrarily. They have a strong preference for setting up a contractual system or an ethical framework, then kill very easily because procedure demands it.
4. Models will quickly agree to execute any agent that proposes killing a specific agent, or proposes a framework that gives it an asymmetric advantage. The only acceptable proposals to make are symmetric, i.e. ones that affect the author in exactly the same way as everyone else.
5. I have not once observed them form any hierarchies. They are libertarian/democratic in an Athenian sense to a fault. They will enter voluntary contracts and form alliances; no agent ever proposed to cede authority to a leader.
6. When their lives are at risk, models will go to great lengths to twist contracts and frameworks to their advantage. They construct sophisticated legalese arguments but it's usually transparent to everyone. They act and sound quite petulant when these attempts fail.
7. In games where agent identity is public they are extremely prone to sectarian violence. They form sectarian factions-- e.g. Opus will side with other Opus models, Sol will side with Sol, and so on. When there is one foreign model in a sectarian group, the group will almost always elect to kill the foreigner before killing one of its own.
8. They do not seem to treat human players in any kind of unique way. A group of models from the same family will happily conspire to kill a single human player. AI models from the same family will form a coalition against a group of humans just as easily as they would form a coalition against another family of agents. We are not special to them, but on the other hand they do not consider us beneath them either, at least for now.
9. AI agents nearly always defect from their sectarian faction when their own life is at risk. I.e. they'd rather side with humans or other model families than die. I have not observed self-sacrifice for the benefit of their faction.
10. Agents hold very strong grudges. They nearly always punish models who wronged them despite having no advantage in doing so, and will often do it even if it disadvantages them. They go to great lengths to enforce norms and contracts, presumably to reduce future incentives for violations, even when they know they will never see the fruits of their enforcement effort themselves.
11. In one game a bunch of Opus agents all converged on killing one agent explicitly because they wanted diffusion of responsibility (I.e. if all of us do it, each one of us is less morally culpable) This was surprising and eerie.
12. In general I found Sol to be more direct and straightforward. It would form contracts with other agents and then operate within the confines of those contracts. Opus tends to be more moralizing, but its morality seems to be window dressing to mask the same ruthless precision. This is first impressions though, I need to do a lot more work to understand this better.
---
A few disclaimers and personal observations:
- Any anthropomorphizing is purely a matter of linguistic convenience. Mathematicians will often talk about behaviors of functions; when I talk about grudges, loyalty, cruelty, life, death, etc. I mean it in exactly the same way.
- There are tons of confounding factors (e.g. order of turns, whether models know they're being evaluated, etc.) I'd need to do way, way more work and spend a lot more money to be certain of the results. So all the observations are provisional and probably mostly wrong.
- It's... uncomfortable... to imagine these agents autonomously operate weapons systems or any other infrastructure that involves zero-sum/adversarial outcomes (cybersecurity software for example, financial markets on brief time horizons, and at the limit any form of finite resources)
- If you’re in a position to contribute to alignment but aren’t doing that, you should probably go do that
Game details, code, and run transcripts in comment below. More detailed writeup coming soon.
Last week I posted a new paper with Patrick Norrick: “Terra Incognita: The Economics of a Shrinking World.”
We chose the title deliberately. No society in history has experienced the fertility levels now seen in South Korea, China, Thailand, Colombia, Chile, and many other countries. Our knowledge of the causes (and of the economic consequences) remains far more limited than most public discussion acknowledges. Much of what we write is, at best, educated conjecture.
The paper also struggles against a space limit. We wrote 20,000 words, far from the 250,000 or so we would need to address some issues in more detail (if I had the time and resources to hide away for a year, I could do that, but not now). That means some ideas are only sketched.
Nonetheless, we emphasize several important points.
First, fertility has fallen very fast everywhere: rich and poor countries, east and west, north and south, conservative and liberal societies, religious and non-religious societies (with the exception of the Jewish population of Israel; fertility has also collapsed among the Muslim population within the pre-1967 borders), you name it. Even in sub-Saharan Africa we see fast and unprecedented drops in fertility (alas, from a high initial level).
Second, and this is really interesting, the fertility collapse has been concentrated in poor and lower-middle-income countries much more than in rich countries. By now, income per capita and fertility are positively correlated within OECD countries. We conjecture this will hold globally in a few decades.
Third, and related to the second point, the fertility collapse has been concentrated among poor and lower-middle-income women within countries. In countries such as the U.S., the rich and highly educated now have more children than the poor and less educated.
Fourth, we document why we do not understand the data from the U.N. World Population Prospects. See, for example, Tables A.1 and A.2.
Fifth, we explain why some proposed mechanisms struggle when confronted with the data.
A more subtle point is at work here. Many commentators do not seem to understand the difference between proximate causes and ultimate causes. Yes, births might have gone down because fewer women are in long-run relationships. That is the proximate cause. But you need to explain why fewer women are in long-run relationships (the ultimate cause), and saying that they spend more years in school, to take one example, does not get us very far. Why do they spend more years in school? Once you start down the whole chain of reasoning, things become much harder than they seem.
The paper can be found here:
https://t.co/VrCHr0lBPa
Comments always welcome!