Thomson

[Thomson @ThomsonYenTY]

Modern businesses run on spreadsheets. Can AI handle financial spreadsheet construction?

We’re excited to share MBABench 2.0: 101 difficult, long-horizon tasks (take up to 33h) across 6 finance areas and most financial model types.

The best AI agent passes only 16% of the tasks

[Thomson @ThomsonYenTY]

@Turnfeed Great point! If by human checkpoint you mean having a human-in-the-loop re-emphasizing the mistakes agents made, then yes it'll likely help the agents. We're actively exploring in this direction actually.

[Thomson @ThomsonYenTY]

Modern businesses run on spreadsheets. Can AI handle financial spreadsheet construction?

We’re excited to share MBABench 2.0: 101 difficult, long-horizon tasks (take up to 33h) across 6 finance areas and most financial model types.

The best AI agent passes only 16% of the tasks

[Thomson @ThomsonYenTY]

MBABench 2.0 would not have been possible without the herculean effort of my collaborators: Julian Poeltl, Patrick Shea, Shubham Anil Varma, Dhruv Rastogi, Zihao Chen, and Mengshi Zhang.

Special thanks to @hsnamkoong for his support and guidance.

[Thomson @ThomsonYenTY]

MBABench 2.0 is a leap forward in financial spreadsheet benchmarking for agents. Read more and access the benchmark here:
Website: https://t.co/c3LATwjgYo
Blog: https://t.co/3zOU5FU5Fg

[Thomson @ThomsonYenTY]

@miniapeur why you write such funny tweets

[Thomson @ThomsonYenTY]

@vincentsunnchen @SnorkelAI this is great for the community!

[Xiao-ke (XK) @xklionfish]

Super excited to share this analysis! If you're interested in robotics data, give our post a read👇

[Thomson @ThomsonYenTY]

@Dhananjay99Ash surprising how OOD is still an issue for envs this simple..

[Dhananjay Ashok @Dhananjay99Ash]

Introducing GameBoyWorlds, a new challenge testbed for self-improvement in embodied video games:

Paper: https://t.co/05Fpom1OM1
Website: https://t.co/qFUZfld80b

[Thomson @ThomsonYenTY]

Getting some very load-bearing NeurIPS metareviews this year

[Snorkel AI @SnorkelAI]

Do agent-built spreadsheets hold up to professional finance standards? MBABench, from @ThomsonYenTY @hsnamkoong and collaborators, evaluates AI agents on end-to-end financial spreadsheet tasks.

See it among 25+ posters at Frontier Data Summit

[Manos Koukoumidis @Koukoumidis]

Enterprise AI is in a wildly paradoxical state 🤔, and we’re reaching the inflection point that will resolve it. 💥

Enterprises want to differentiate with AI, yet rent the same intelligence as their competitors.

Their workflows and expertise are highly specialized, yet they rely on generic models built to be good at everything.

They worry about AI costs, yet pay premium prices for massive models where only a fraction (1%) of the intelligence is relevant to their task. 💸

And they demand control and sovereignty, yet rent the intelligence becoming core to their business.

This is not a sustainable equilibrium.

The next era of enterprise AI is specialized intelligence companies build, own, and compound. And we are at the inflection point of this transition.

That’s the bet we made when we started @oumi_ai two years ago.

Today we’re closing the loop: Oumi can now not only automatically build your specialized AI models, but also deploy them into production, learn from their production experience, and continuously improve them.

The intelligence that your business runs on, becomes your differentiator. Your compounding advantage.

The winners of the next AI era will turn their own data, expertise, and experience into specialized intelligence that nobody else can rent.

Don’t rent your AI. Build it. Own it. Compound it.

[Thomson @ThomsonYenTY]

Grateful to the team that built this @cool_gear9 @iris_y_meng @siridu21 @amaster97 @aliwka_bb, Julian Poeltl, Joshua Fan, Sigrid Liu and Haoyang Liu. This would not have been possible without the countless hours our experts have spent on refining the criteria and reviewing agents’ attempts. Also very thankful to @hsnamkoong and Daniel Guetta for their guidance.

[Thomson @ThomsonYenTY]

Today we're releasing MBABench 1.0: the first, spreadsheet-focused benchmark evaluating AI agents on building entire workbooks to professional standards.

In the real-world, MBAs must produce complete, well-organized artifacts ranging from spreadsheets to presentations. These deliverables are then reviewed and revised by multiple stakeholders ranging from managing directors to client analysts, so beyond accuracy, they must meet various quality standards specific to the industry.

We evaluated 10+ frontier models on end-to-end spreadsheet tasks, and found agents still fall short of the standards MBA grads are held to. (1/n)

[Thomson @ThomsonYenTY]

Read more about MBABench here:
Blog post: https://t.co/0OjrtDpDtC
Website: https://t.co/2guq6AVSdv
Code: https://t.co/PPtfGspJDK
Paper: https://t.co/l4QknKDDtO