Annalisa Barbara

CS PhD Student @ETH | RL, game theory & alignment. Views are not my own.

[Yi Ma @YiMaTweets]

Just came back from ICML. Gave a keynote at the Foundation of Deep Generative Models Workshop, in which I stated that Intelligence should be a scientific subject, arguably much more significant than Physics. It is high time we study it with the same scientific methodology and mathematical rigor as modern physics, instead of always at the level of being empirical, meta physical, meta mathematical, philosophical or speculative... This remains as the biggest opportunity ever for young scientists. To my knowledge, this is not the focus of any of the frontier "AI" companies.

[Annalisa Barbara @annalisab]

It is interesting to reflect on the spatial distribution of topic areas at the ICML poster sessions. In particular, from the far right edge to the far left edge, we have:

Theory → General ML → Optimization → Probabilistic Methods → Social Aspects → Deep Learning → Applications → Reinforcement Learning

What stuck with me is not only that Theory and RL posters are at the very edges, but also that they are as far away from each other as physically possible. So if you want to see posters from both areas, you basically need to walk through hundreds of posters in the middle, which is maybe ICML’s way of forcing some interdisciplinary cardio.

I wonder if this is just a random allocation, or indeed some kind of political choice.

[Lorenzo Tarricone (@ICML🇰🇷) @LTarricone01]

Excited to share our work on GPU parallelisation for designing large protein nanoparticles #ICML2026 🔬

We enable larger de novo protein nanoparticle design by distributing attention memory costs across multiple GPUs.

📍 Hall D2
🗓 Fri 10/07/26
🕒 2:45–3:30 pm

[Annalisa Barbara @annalisab]

Joint work with Riccardo, Martino & Andrea, from my Italian alma mater:
𝑂𝑝𝑡𝑖𝑚𝑎𝑙 𝑅𝑎𝑡𝑒𝑠 𝑓𝑜𝑟 𝐹𝑒𝑎𝑠𝑖𝑏𝑙𝑒 𝑃𝑎𝑦𝑜𝑓𝑓 𝑆𝑒𝑡 𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑖𝑜𝑛 𝑖𝑛 𝐺𝑎𝑚𝑒𝑠
https://t.co/H8hAUWwOxg

[Annalisa Barbara @annalisab]

Wandering around Seoul next week for ICML. Happy to talk about, in decreasing order:
🍜 Korean food
🐢 What questions are worth spending 4 years of a PhD on (or however long we have before AGI)
✨ Recent 𝐬𝐩𝐨𝐭𝐥𝐢𝐠𝐡𝐭 𝐩𝐚𝐩𝐞𝐫 on inverse game theory!

[Annalisa Barbara @annalisab]

Also increasingly convinced zero-sum is not a great notion for adversarial games and that this needs rethinking.

[vivek @itsreallyvivek]

https://t.co/ZhOgyq7Vgn

[RL Theory Virtual Seminars @RLtheory]

The RL Theory Seminar returns in June, one hour earlier. We're broadening the scope a bit, including RL for LLMs and robotics, and featuring a few more applied papers alongside foundations-oriented work. Please share widely!

We start next Tuesday with Daniel Russo.

[Andrew Ng @AndrewYNg]

There will be no AI jobpocalypse.

The story that AI will lead to massive unemployment is stoking unnecessary fear. AI — like any other technology — does affect jobs, but telling overblown stories of large-scale unemployment is irresponsible and damaging. Let’s put a stop to it.

I’ve expressed skepticism about the jobpocalypse in previous posts. I’m glad to see that the popular press is now pushing back on this narrative. The image below features some recent headlines.

Software engineering is the sector most affected by AI tools, as coding agents race ahead. Yet hiring of software engineers remains strong! So while there are examples of AI taking away jobs, the trends strongly suggest the net job creation is vastly greater than the job destruction — just like earlier waves of technology. Further, despite all the exciting progress in AI, the U.S. unemployment rate remains a healthy 4.3%.

Why is the AI jobpocalypse narrative so popular? For one thing, frontier AI labs have a strong incentive to tell stories that make AI technology sound more powerful. At their most extreme, they promote science-fiction scenarios of AI “taking over” and causing human extinction. If a technology can replace many employees, surely that technology must be very valuable!

Also, a lot of SaaS software companies charge around $100-$1000 per user/year. But if an AI company can replace an employee who makes $100,000 — or make them 50% more productive — then charging even $10,000 starts to look reasonable. By anchoring not to typical SaaS prices but to salaries of employees, AI companies can charge a lot more.

Additionally, businesses have a strong incentive to talk about layoffs as if they were caused by AI. After all, talking about how they’re using AI to be far more productive with fewer staff makes them look smart. This is a better message than admitting they overhired during the pandemic when capital was abundant due to low interest rates and a massive government financial stimulus.

To be clear, I recognize that AI is causing a lot of people’s work to change. This is hard. This is stressful. (And to some, it can be fun.) I empathize with everyone affected. At the same time, this is very different from predicting a collapse of the job market.

Societies are capable of telling themselves stories for years that have little basis in reality and lead to poor society-wide decision making. For example, fears over nuclear plant safety led to under-investment in nuclear power. Fears of the “population bomb” in the 1960s led countries to implement harsh policies to reduce their populations. And worries about dietary fat led governments to promote unhealthy high-sugar diets for decades.

Now that mainstream media is openly skeptical about the jobpocalypse, I hope these stories will start to lose their teeth (much like fears of AI-driven human extinction have).

Contrary to the predictions of an AI jobpocalypse, I predict the opposite: There will be an AI jobapalooza! AI will lead to a lot more good AI engineering jobs, and I’m also optimistic about the future of the overall job market. What AI engineers do will be different from traditional software engineering, and many of these jobs will be in businesses other than traditional large employers of developers. In non-AI roles, too, the skills needed will change because of AI. That makes this a good time to encourage more people to become proficient in AI, and make sure they’re ready for the different but plentiful jobs of the future!

[Original text in The Batch newsletter.]

[Dwarkesh Patel @dwarkesh_sp]

It's very interesting that cryptographic protocols and neural networks have the same high-level architecture (where they jumble information as it moves sequentially across many layers).

This is the result of a convergent evolution - cryptographic protocols need every output bit to depend on every input bit in complicated ways, and similarly, NNs need output to make connections between inputs.

But they're in some sense doing opposite things. While cryptographic protocols take something which has a lot of structure and make it seem indistinguishable from random, NNs take something which may look random and extract structure from it.

Much more on this idea in the full episode with @reinerpope

[Richard Sutton @RichardSSutton]

RL Ethics has a Predictive Semantics

I would like to try to explain the view of ethics and values that arises from my research in reinforcement learning in simple, layman’s terms that are accessible to all.

Reinforcement learning agents seek to maximize their reward over time, where reward is essentially pleasure minus pain. This is not quite hedonism, because the maximization takes into account all the consequences, long-term as well as short. A reinforcement learning agent might endure pain to get a larger pleasure later, or forego an immediate pleasure if it stored up later, greater pain. Formally, reward is a number at each time step, and the reinforcement learning agent seeks to maximize value—the sum of the rewards at future time steps. (This could be defined precisely with some math.)

The assignment of rewards to time steps is a free choice that defines the agent’s goal; different agents could have different rewards, and there is no basis (yet) for preferring one set of rewards over another. Value though is a different matter. Given a world and a way of generating rewards, the true values at each time step are fully determined. The rewards are primary, dependent on nothing else, whereas the values are secondary, following from the rewards (and the dynamics of the environment). In decision making, the agent should make the choice that leads to highest immediate value, not highest immediate reward.

If rewards are arbitrary, values follow from the rewards, and correct behavior follows from the values, then all seems straightforward. What about all the complexities and controversies of ethics? Some of these are still present, arising because the values, though well defined, are initially unknown and can be difficult to calculate or learn.

If the agent has knowledge of the world, then it may be able to calculate the values, but to do so exactly generally requires too much knowledge, computation, and memory. In practice, in new situations the calculation must be done partly at decide time, and cannot be done to completion without slowing down action selection too much.

In the absence of knowledge and computation, but given a generous allocation of memory and time, the agent can alternatively learn the values, again approximately. It is common for the agent to store an approximation to the world’s state’s values, and then to gradually improve these approximations—these predictions of subsequent rewards—by further experience. The stored approximate values are immediately available estimates of the desirability of situations; they are directly analogous to our intuitive sense of good and bad. They are ready for immediate use, but may only be rough approximations to the true values. They may be made more accurate with calculation (if the agent has knowledge) or learning (with more experience).

This completes the explication of the value system of the individual. Next we will go on to consider the value systems of groups. But the individual forms such an essential foundation that is never replaced, so let’s dwell on it a moment longer by reviewing its stark tenants: Each agent wants to get pleasure (reward) from the world. Pleasure is built-in to the agent and obvious when it happens, but when it will happen depends on the world and must be learned or calculated—and the world is too complex for either of these methods to yield answers that are completely correct. That is, every state of the world has a real, objective value (the amount of pleasure that will follow it), but estimates of its value are subjective. Forming better value estimates is a major cognitive task. They are a key intermediate step towards getting more pleasure from the world. Agents work on this all the time. It determines what they do.

If an agent lived alone, then this would be the end of our discussion of values and ethics. But people are not solo agents. Peoples’ worlds are comprised, in part, of other people, and this has many impacts of their attempts to estimate value and obtain reward. They live within groups of agents with whom they interact frequently and whom are major determinants of their success is obtaining reward. And thus, to achieve our reward, each of us must take into account, as best we are able, the rewards and values of those around us.
…

The most important insight is that it's alright, and perhaps obligatory, for the ultimate value to be hedonic (based on reward), as long as it is not "selfish" (disregarding the impact on others). The ultimate meaning of something being good, or right, or ethical, or moral, is that it will probably have a good outcome for the individual. Whether it will or not is extraordinarily difficult to calculate, so instead we use heuristics—approximations using features of a situation. The mistake is to think that those features are definitional rather that approximate predictive. The real definitional meaning of good is that it turns our well for us on average.

[Dwarkesh Patel @dwarkesh_sp]

Did a very different format with @reinerpope – a blackboard lecture where he walks through how frontier LLMs are trained and served.

It's shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk.

It’s a bit technical, but I encourage you to hang in there - it’s really worth it.

There are less than a handful of people who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him.

Recommend watching this one on YouTube so you can see the chalkboard.

0:00:00 – How batch size affects token cost and speed
0:31:59 – How MoE models are laid out across GPU racks
0:47:02 – How pipeline parallelism spreads model layers across racks
1:03:27 – Why Ilya said, “As we now know, pipelining is not wise.”
1:18:49 – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal
1:32:52 – Deducing long context memory costs from API pricing
2:03:52 – Convergent evolution between neural nets and cryptography

[Peyman Milanfar @docmilanfar]

when does a walk become a hike

[Jonathan Gorard @getjonwithit]

I think one of the conclusions we should draw from the tremendous success of LLMs is how much of human knowledge and society exists at very low levels of Kolmogorov complexity.

We are entering an era where the minimal representation of a human cultural artifact... (1/12)