Jiayu Liu

[Keyang Xuan @keyang_xuan]

AI evaluation is entering an interactive benchmark era.

Across tool-use agents, web/OS benchmarks, multi-agent systems, and reliability evaluations, interaction is becoming central to how modern AI systems are tested.

But the field risks adding interaction faster than it develops the scientific principles for evaluating interaction.

Our position:
Interactive evaluation is not just longer tasks, tool use, or multi-turn interaction.
It requires a design science for mapping trajectories to valid evaluative claims.

📄 https://t.co/lKGuDOuBZy
💻 https://t.co/LkadiPYnnw

[Jiayu Liu @Liujy03]

RT @chongrui28836: Introducing Auto-Dreamer 🧠💤

A research counterpart to @AnthropicAI's "Dreaming" for Claude Managed Agents, exploring th…

[Anders Giovanni Møller @AndersGiovanni]

👩🏼‍💻 Real or Robotic? 🤖 Can LLMs accurately simulate qualities of human responses in dialogue?

Human conversations with LLMs are great for assessing the capabilities of LLMs. But having lots of folks chat with LLMs is challenging (💰⏳🕵️). Could we have another LLM *simulate* being a human talking to an LLM as a substitute?

In our new preprint, we test whether models can roleplay as the human in human-LLM conversations. Using the WildChat dataset and 100K+ simulations we test how well these LLM responses actually mimic with human ones. Our study spans 🇬🇧 English, 🇨🇳 Chinese, and 🇷🇺 Russian, using 21 linguistic metrics like lexical, semantic, syntactic, and stylistic features.