Manuel Mueller-Frank

Economics Professor at IESE Business School. Decentralization Maximalist. Game Theorist.

[Mallesh Pai @malleshpai]

The bet is simple: higher-order beliefs can help separate “popular with this rater/model” from “robustly recognized as good.”

Could mean better multi-model aggregation, cleaner RLHF/DPO labels, better LLM-as-judge evals, and smarter triage of examples for human review.

We would love to collaborate with people running evals or post-training pipelines to see if/ how these ideas have legs. Reach out!

[Mallesh Pai @malleshpai]

A few natural places to try it:

Training-free aggregation: ask models for answers + predictions of what other good models would answer.

Post-training: ask raters for preferences + predictions of other raters’ preferences.

Evals: ask judges for scores + predictions of panel scores.

[Mallesh Pai @malleshpai]

One direction I’m particularly excited by: using PMBA-style aggregation to improve our chat AI overlords.

A lot of the work in refining AI models has the same structure: many noisy judgments, no obvious ground truth, and a lot riding on how we aggregate them.

[Mallesh Pai @malleshpai]

A few people have said some version of: “Isn’t this just Bayesian Truth Serum / Surprisingly Popular / higher-order beliefs?” Basically: yes! That’s the lineage, and we cite it. We wrote this because we think there’s still a lot of juice here!

What we add is mostly two things:
(1) moving beyond binary questions, since lots of real applications have many possible answers; and
(2) showing that what can look like a “too clever by half” mechanism is really just linear regression.
That second part is useful because regression gives you applied machinery people already know how to use: uncertainty, standard errors, confidence intervals, ways to think about noisy reports, etc.
The goal is to make higher-order-belief aggregation less of a cute trick and more of a usable empirical tool.

[Alex Imas @alexolegimas]

This paper is simply wonderful. Shows you how simple wisdom of the crowds can go terribly wrong, and how just one more question (asking people what they think others think) does a much better job. Simple beautiful method for an age old problem.

[Mallesh Pai @malleshpai]

Updated Paper 🚨 (with Yi-Chun Chen and @ProfMMF)

The “wisdom of the crowd” is one of the oldest and prettiest ideas in social science: Ask enough people, average their answers, and the random errors cancel out. Underlies Aristotlean philosophy, Galton and the Ox, and these days, Prediction Markets.

Beautiful, magical when it works, but sometimes… completely wrong! 🧵

[Manuel Mueller-Frank @ProfMMF]

@WPReadingClub Thank you. That's why we are all here. Regarding the paper, I would be happy to present it in your setting.

[Manuel Mueller-Frank @ProfMMF]

@simonkim_nft Nice article but is it accurate to say that transactions involving stablecoins issued by a central entity (Tether, Circle) execute autonomously when the centralized issuer has the power to censor transactions/freeze accounts?

[Manuel Mueller-Frank @ProfMMF]

@JasonYanowitz CT shares a common prior that is heavily tilted towards bigger brand and most respondents have no private information. You could dramatically increase the informativeness of the exercise by eliciting higher-order information. Look up the Surprisingly Popular and PMBA procedure.

[Manuel Mueller-Frank @ProfMMF]

@therabbithole Up only

[Manuel Mueller-Frank @ProfMMF]

@VirajNad @SiyangXiong Thank you Viraj. Looking forward to your comments.

[Manuel Mueller-Frank @ProfMMF]

🚨New paper alert🚨
Oracles perform well when reporting agents commonly know the state (say ETH/$), i.e., all know the state, know that all know it and so on. In our new paper, @siyangxiong and I analyze the robustness of oracles when common knowledge is slightly disrupted. 1/n

[Manuel Mueller-Frank @ProfMMF]

Our results highlight a novel tension between decentralization and robustness for simultaneous oracle mechanisms: they cannot satisfy both properties. Sequential mechanisms can.
Full paper here:
https://t.co/nbycOkOALF
n/n

[Manuel Mueller-Frank @ProfMMF]

Not all is lost, however. We construct a sequential direct voting mechanism that satisfies strong (sequential-)continuous implementation and that is not subject to single-deviation. 6/n