Hong Namkoong

Assistant prof @ Columbia DRO

[Thomson @ThomsonYenTY]

Modern businesses run on spreadsheets. Can AI handle financial spreadsheet construction?

We’re excited to share MBABench 2.0: 101 difficult, long-horizon tasks (take up to 33h) across 6 finance areas and most financial model types.

The best AI agent passes only 16% of the tasks

[vincent sunn chen @vincentsunnchen]

Today, we’re expanding @SnorkelAI’s Open Benchmarks Grants by 10x to a $30M commitment:
- Funding a more diverse, robust, and continuously-updated ecosystem of open benchmarks
- Launching the Open Benchmarks Red Team to continually test and strengthen them
- Introducing the Snorkel Research Fellowship to support independent contributors developing new evaluation methods

Since our launch earlier this year, we’ve been humbled to partner with the teams behind Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, Agents’ Last Exam, Continual Learning Bench, Senior SWE-Bench, SlopCodeBench, and more. Alongside these teams, we’ve helped design evaluation methodologies, develop task construction pipelines, and scale quality control. OBG-funded benchmarks have appeared on the latest model cards from every major frontier lab; have helped to index and measure frontier progress and alignment; and have contributed to guiding the frontier of AI development.

Benchmarks have always been guideposts for—and drivers of— AI progress: setting a metric and then “hillclimbing” against it is the core of how AI works! But when benchmarks fall behind frontier capabilities, our ability to evaluate and align these systems does as well. They become too simple, static, or correlated, and the risk of “benchmaxxing” increases.

We need more diverse, robust, and continuously-updated benchmarks, developed by a broader ecosystem of researchers and domain leaders.
We’re incredibly excited to accelerate frontier evaluation with Open Benchmarks Grants— raising the ambition and investment in robust, open, and independent measurement. Read our full announcement and apply here: https://t.co/E5kfZQ0Im5

[Dhananjay Ashok @Dhananjay99Ash]

Introducing GameBoyWorlds, a new challenge testbed for self-improvement in embodied video games:

Paper: https://t.co/05Fpom1OM1
Website: https://t.co/qFUZfld80b

[Hong Namkoong @hsnamkoong]

I'm excited to release MBABench, a new eval for nuanced judgement-driven tasks that MBAs perform. Being able to interact with professionals in NYC has provided many unexpected learnings on the impact of AI in white-collar work, and this project reflects this rich confluence!

[Thomson @ThomsonYenTY]

Today we're releasing MBABench 1.0: the first, spreadsheet-focused benchmark evaluating AI agents on building entire workbooks to professional standards.

In the real-world, MBAs must produce complete, well-organized artifacts ranging from spreadsheets to presentations. These deliverables are then reviewed and revised by multiple stakeholders ranging from managing directors to client analysts, so beyond accuracy, they must meet various quality standards specific to the industry.

We evaluated 10+ frontier models on end-to-end spreadsheet tasks, and found agents still fall short of the standards MBA grads are held to. (1/n)

[Percy Liang @percyliang]

For the next Marin model, we are putting together a new data mix. Currently we have 18T tokens, but could use more. So if you are sitting on some secret stash of high quality tokens, please let us know! Pre-training, mid-training, SFT data all welcome.

[Fanny Yang @FannyYangETH]

📢We're hiring! The Statistics Machine Learning group @ETH Zurich🇨🇭is looking for 𝙥𝙤𝙨𝙩𝙙𝙤𝙘𝙨 (fully-funded) with a strong math background & keen interest in reliable ML👇
📅Start date: 2026, flexible
➡️Apply: https://t.co/GIlnq3uD0r
#Postdoc #Statistics #ML #Switzerland

[tom zollo @SquareZollo]

How can we train LLMs to:
1) Accurately quantify uncertainty
2) Dynamically ask the right questions to reduce it?

Excited to share our new #ICML2025 work on this topic!

[Hong Namkoong @hsnamkoong]

@nicochristie This looks cool! Pls share an invite code.

[Tony Chen @tonychenxyz]

Introducing our #NeurIPS2024 Dataset & Benchmark paper: QGym -- a scalable framework for simulation and benchmarking of queuing network controllers.

Paper: https://t.co/vxMJMjl7pv
Code: https://t.co/iDvxdIlwnV

(0/n)

[Jiashuo Liu @liujiashuo77]

Time to rethink ML robustness! Analyzing 7,650 source-target shift pairs on ACS data, we found Y|X-shifts prevalent in tabular distribution shifts, often driven by missing features or confounders. Our WhyShift benchmark is the first to explicitly address specific shift patterns.

[Jiashuo Liu @liujiashuo77]

New work: LLMs for tabular data!

We show LLM embeddings effectively tackle Y|X shifts, significantly boosting model adaptability under distribution changes. Plus, they provide a unified interface for easily integrating extra domain information to further enhance performance!

[Andrew Siah @theandrewsiah]

[New research direction on Personalized LLMs] For complex tasks where users have vastly different needs, LLMs should adapt to each individual user.

https://t.co/siOJNyC6xc 🧵

[Shiori Sagawa @shiorisagawa]

Join us at the NeurIPS Workshop on Distribution Shifts (DistShift) tomorrow!

When: Friday, Dec 15, 9am-5pm
Where: Room R06-R09
Website: https://t.co/CgZD0PsIWW
Virtual site: https://t.co/eggFNJm2MB

[Hong Namkoong @hsnamkoong]

Excited to give a tutorial tomorrow on data heterogeneity and distribution shifts at #NeurIPS2023. Despite the marvelous rates of progress, AI models still fail to generalize in out-of-distribution environments. Yes, even largest models trained on the entirety of the internet!

[Hong Namkoong @hsnamkoong]

cc: DistShift organizers:
@yoonholeee @shiorisagawa @BeccaRoelofs @FannyYangETH. M Sugiyama @jacobeisenstein @PangWeiKoh @tatsu_hashimoto