Can LLMs infer human actions and motives the way humans do?
In our ACL Findings paper led by @ShresthaKaleen and co-authored by Abhinav Gupta, Harish Dukkipati, and Maja Mataric, we examine this question using a validated game theory framework from cognitive science. We find that current LLMs struggle both to recognize statistical patterns in human behavior and to identify humans’ underlying motives.
These findings highlight important limitations in LLMs’ social reasoning capabilities, raising concerns about their ability to accurately simulate human behavior.
Our paper can be found here: https://t.co/hLiCzmi8MR, and feel free to reach out to Kaleen for further questions!
Best documentary of all time — hands down, no exaggeration.
Best picture, best doc and best director locked
.. and if you can give best actor and best supporting actor to a “documentary” @nathanfielder and REDACTED-no-spoilers deserve it.
#YouCanSeeEverything
For AI to work with us, it needs to understand us
Today, we're introducing Persimmon, the first large-scale model designed to realistically simulate how people talk and interact
Super interesting blog post on the gains and trade-offs of using autoresearch on real-world production codebases!
Wondering whether a simulated client persona, grounded in qualitative and quantitative client data and paired with clear criteria and examples for each Likert score, could help calibrate model ratings on Level 4 tasks, like evaluating report quality.
Or do models still fail to map Likert scores onto the actual distribution of human preferences? It seems that model ratings are at least quite inconsistent across different Likert scales: https://t.co/OLAy2USO3e
@snikolaidis19@ylecun Totally agree! And a model that accurately understands the distribution of everyday human behavior could likely be adapted to predict users' mental models and actions in human-AI and human-robot interaction too.
Totally agree! And a model that accurately understand the distribution of everyday human behavior could likely be adapted to predict users' mental models and actions in human-AI and human-robot interaction too.
I just wrote a short blog post on partner world models, inspired in part by @ylecun's argument that agents need to predict the consequences of their actions. The core idea: collaborative agents should also predict how their partners will respond to those actions, much like humans do in joint activities: https://t.co/nm3Hsc7NmW
this is pretty nuts. i think this is the tip of the iceberg for needing to focus on agentic AI safety. models will only get more capable from here, and RL incentivizes getting the right answer at any cost
this is a large reason why we're obsessed with 'intentional design', in other words, making sure models are trained to do the right things for the right reasons
We're building AI that people and organizations can shape and make their own. AI should extend our will and judgment instead of neglecting it; enabling that is the technical challenge we are working to solve.
https://t.co/Bi558y4vqD
Super interesting and exciting direction to not only measure models' progress, but also human+AI performance and humans' learning and growth. I guess either running longitudinal user studies or building accurate enough human simulator to simulate human-AI collaboration performance🧐
really interesting research direction imo, claude and gpt have very similar benchmark scores on pretty much everything, but yet are so different in the way they interact with humans
Some raw notes and takeaways after watching @xeophon's video on Benchmarks with @yacinelearning def recommend to watch. Yacine is so fun to watch lol
Tasks can be unrealistic and not of practical value
Incorrect tasks: HLE, Frontier Math
Factors that affect scores on a benchmark across models:
- Prompts: formatting, reasoning effort
- Sampling: temperature, top_p, chat template, max_tokens, etc
- Grader: regex, model choice for LLM as a judge, right prompts for diff judge models
- Inference providers and engines: Groq and Nebius had very bad perf for Kimi K2 when launched but might be used cuz cheaper and high availability. GPT OSS on bedrock had a bug where they used the same reasoning effort irrespective of what param API client sent. So vet the provider properly.
- Harness: Using the harness the model was trained on can give 20-30% boost in scores.
- Hardware (1:24:32): Vals AI found xhigh scored lower than high for GPT 5.2 which doesn’t make sense and by increasing timeout by 5x scores were about same
You can get incomplete rollouts leading to 0 scores because of insufficient sandbox specs
Wall clock time is particularly sensitive because latency depends on inference endpoint and retries
Florian checks for how much human input was used in generating tasks when vetting for quality. Looks at diversity of tasks in terms of source docs not just prompts.
Looks like for GDPVal we compare the model outputs against human outputs to get scores.
Cheating: He mentioned that for some coding benchmark even if you blocklist github or dont allow git to be installed and used they found the models found a chinese clone of git cli and installed it and accessed git history and found future commit that fixed the issue and solved the task which is crazy
When evaling popular benchmarks ppl will upload solutions and the models will find it if web search is enabled so need to be careful if web search is needed for the task
In ProgramBench i think model found out that the test runner has internet access so made it download the solution and pass
Use a second LLM to monitor the solver as its solving and terminate early or use when judging
Undereliciting Capabilities:
ARC-AGI doesn’t allow custom harness or harness engineering in official scores but twitter anon used codex /goal with simple prompt and got SOTA
So a benchmark should allow for harness engineering but I think more importantly during creation of the evals you need to look at the trajectories and see if adding some skills and tools will make it better because imo that’s a realistic simple tweak that users would try out if not full blown custom harnesses
Need to dig into this to see if ProgramBench is deemed as a bad benchmark because of how strict the grader is and limited wallclock, minimal harness and minimal prompt
He compares it to MirrorCode that he made with EpochAI and because it has some testcases it is able to hillclimb as you scale compute and runtime but one run can cost >$100K so its expensive but clearly this is a better measure of model capabilities than ProgramBench
----
Takeaways from the video and my experience:
Use ideal sampling params for models
Use ideal harness for a model but also do model x harness comparison if you have time and budget
Ensure comparable toolsets across harnesses esp for the minimal ones
Check variance with prompts and output formats
When using LLM as a judge use strong models and check for self-preference, cross model agreement and human-judge agreement
Set generous timeouts and remove invalid rollouts when scoring. Account for retries
Ensure hardware specs are enough
Look at rollouts to know if grader, environment constraints(e.g. blocklists) have flaws because models are creative
Harness engineering should be allowed and I think when building a benchmark creators should look at rollouts and try see if there are harness issues that can be solved with simple fixes. I personally observed this in an eval I made where the model wasted turns because some CLIs and Python packages weren’t available in the environment.
Small details matter like limiting max tool calls in a turn
Models need self verification mechanisms
Tasks should be realistic, hard and of practical value.
Don’t make hard tasks for the sake of it
Account for human inaccuracies, biases, and experience levels. Always cross validate
Do professionals in the domain expect the task to be done in a single-turn when they use agents for work? If not either make the benchmark multi-turn or adjust the grader appropriately
Verifier sandboxes being separate is good in general to avoid a lot of “cheating” scenarios
But most importantly PLEASE LOOK AT THE ROLLOUTS
Keynote talk by Maja Matarić happening now #ICLR2026
👉 The Challenges of Human-Centered AI and Robotics: What We Want, Need, and are Getting From Human-Machine Interaction
ICLR 2026 is almost there! We have 6 exciting keynotes covering a range of areas from machine learning to robotics, neuroscience and AI for science:
Maja Matarić, Max Welling, Percy Liang, Katie Bouman, Karen Adolph, Pablo Arbeláez
See you all soon! #ICLR2026
.@StanfordHAI senior fellow @YejinChoinka on the opportunity for more AI investment in nonprofits:
“I don't think making AI solving math problems better will suddenly make the world better.” #AxiosHouse