Ken Gu

PhD Student @uwcse | Student Researcher @GoogleResearch | AI for Data-Driven Science | Previously @MSFTResearch, @UCLA

[Ken Gu @kenqgu]

🚨Are LLMs truly ready for autonomous data science?

Real-world data is messy—missing values, outliers, inconsistencies—and if not handled properly, can lead to wrong conclusions.

🌟We introduce RADAR, a benchmark evaluating whether LLMs can handle imperfect tabular data. 🧵

[Alex Shaw @alexgshaw]

Today, we’re announcing the next chapter of Terminal-Bench with two releases:

1. Harbor, a new package for running sandboxed agent rollouts at scale
2. Terminal-Bench 2.0, a harder version of Terminal-Bench with increased verification

[Ken Gu @kenqgu]

Even as agents gain greater autonomy, it’s critical to consider human collaboration in agent design and evaluation. This work is an important step in this direction. Check it out!

[Shannon Shen @shannonzshen]

Today's AI agents are optimized to complete tasks in one shot. But real-world tasks are iterative, with evolving goals that need collaboration with users.

We introduce collaborative effort scaling to evaluate how well agents work with people—not just complete tasks 🧵

[Ken Gu @kenqgu]

@harsh3vedi Thank you Harsh! I really appreciated your feedback on this project 👏

[Ken Gu @kenqgu]

📃 Paper: https://t.co/ITwP4YdtDG
💻 Eval Code: https://t.co/Wx3bLQxwyi
🤗 Data and Tasks: https://t.co/OE0HHJZfcp

Huge thanks to my co-authors @advaitmb, @Mike_A_Merrill, @cervisiarius, @xliucs, @danmcduff, and @timalthoff 🎉

[Ken Gu @kenqgu]

True intelligence = reasoning about new information, not memorized facts.

How can we scalably create benchmarks that are completely novel yet have known answers?

Meet SynthWorlds, an eval & data-gen framework to disentangle reasoning and knowledge⬇️🧵

📄https://t.co/ITwP4YdtDG

[Ken Gu @kenqgu]

Lastly, our framework is fully automatic 🤖!

SynthWorlds scales to new worlds and tasks beyond our initial dataset📈

Generate novel worlds, facts, and tasks to benchmark reasoning. No human labeling required.

Perfect for synthetic data and testing agentic/long-context models🏄

[Tim Althoff @timalthoff]

(please reshare) I'm recruiting multiple PhD students and Postdocs @uwcse @uwnlp
(https://t.co/I5wQsFnCLL). Focus areas incl. psychosocial AI simulation and safety, Human-AI collaboration.

PhD: https://t.co/ku40wCrpYh

Postdocs: https://t.co/K9HUIPJ5h6

[Ken Gu @kenqgu]

📃 Paper: https://t.co/6AQafUHoUV
💻 Code: https://t.co/yYX5h3YiJw
🤗 Data: https://t.co/MWDsVmlRmL

This was a team effort with @k8_lin_, @Yahskapar, @kazemi_sm, @kmr_ayush, @yang_yuzhe, @hamidpalangi, @Orson_Xu, @danmcduff, @timalthoff, @xliucs, and other amazing co-authors.

[Ken Gu @kenqgu]

This framework lets us systematically generate tasks across table sizes and imperfect table variants.

In total, RADAR includes 2,980 table–query pairs, 53 tasks, 6 artifact variants (including 1 clean), 3 column number variants, and 4 table sizes: 2K, 4K, 8K, and 16K tokens.

[Ken Gu @kenqgu]

💡Our results highlight key implications for deploying LLM agents: when to allocate test-time compute, how to efficiently represent data tables for detecting imperfections, and how to coordinate code execution with table inspection.

Additional results and details in the paper!