Hongbin Na | 那洪彬

Second Year MRes student @UTSEngage. Prev @XJTLU, @LivUni. AI for Social Good. #NLProc.
INTJ 5w4

[Stephan Rabanser @steverab]

📢 Just wrapped up some cool agent evaluation work for ICLR? Or are you simply interested in learning more about the evals landscape? Come to our Agent Evaluation Science Fall Symposium in New York City on November 20, 2026 where we bring together researchers & practitioners!

The day will focus on how agents should be scientifically evaluated, with topics spanning measurement and benchmark design, reliability, safety, and real-world performance. If you have some exciting work in this space, please consider submitting it for poster presentation!

Abstract registration deadline: October 20
Submission deadline: October 25
OpenReview: https://t.co/3R6N0UI3KS

We are also excited to welcome a strong lineup of speakers, with more to be announced. Confirmed so far: @yoavartzi (Cornell Tech), @jaseweston (Meta AI), @nouhadziri (Cohere Labs), @PeterHndrsn (Princeton University), @walesalaudeen96 (Microsoft Research), @criticalneuro (Microsoft Research), and @saurabhjha2010 (IBM Research).

Important: You do not need to submit work in order to join! Just register through the Luma link below. If space becomes limited however, we will prioritize authors with accepted submissions.

Register: https://t.co/u3VmVzzpYD
Details: https://t.co/v0D95KuP4o

Huge thanks to the entire organizing team, including @yininghua, @HongbinNLP, @manuelcherep, @dubrovnik07, @HuaYilun, @ishapuri101, @liuzhuang1234, Lori Chibnik, @vbingliu, @CsabaSzepesvari, @PeterStone_TX, @sirrice, @ang3linawang, and @nikhilsinghmus for putting this together!

Feel free to reach out with any questions and we hope to see many of you there!

[Yulong Chen @Yulongchen1010]

Can LLMs learn a new language using only a grammar book and a dictionary like how human adult L-2 learners do?

Check our in-progress paper! The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang https://t.co/DWq4dRwvD7

[Omar Shaikh @oshaikh13]

BREAKING NEWS! Most people aren’t prompting models with IMO problems :)

They’re prompting with tasks that need more context, like “plz make talk slides.”

In an ACL oral, I’ll cover challenges in human-LM grounding (in 60K+ real interactions) & introduce a benchmark: RIFTS.

🧵

[Dora Zhao @dorazhao9]

While we’re building amazing new human-AI systems, how do we actually know if they work well for people?

In our #ACL2025 Findings Paper, we introduce SPHERE, a framework for making evaluations of human-AI systems more transparent and replicable.
✨https://t.co/afVCweAqZU

[Kobi Hackenburg @KobiHackenburg]

Today (w/ @UniofOxford @Stanford @MIT @LSEnews) we’re sharing the results of the largest AI persuasion experiments to date: 76k participants, 19  LLMs, 707 political issues.

We examine “levers” of AI persuasion: model scale, post-training, prompting, personalization, & more
🧵

[Hongbin Na | 那洪彬 @HongbinNLP]

Excited to present our survey on LLMs in Psychotherapy at #ACL2025!

Let's discuss a coherent path forward! Find me at my poster:
📍Location: Hall 4/5
🗓️Session: IP-Poster Session 5, Mon, July 28, 18:00-19:30

#LLM #AI #MentalHealth #Psychotherapy

[Dongxi 东锡 NLP @dongxi_nlp]

Kimi K2 的一大亮点，是将文本任务里基于 token 的处理思路，成功迁移到 Agentic 场景中的 tool-call 级别：在 Agentic 任务中，tool call 就相当于“行动 token”。

什么意思呢？解释如下：

在文本任务中:
CoT 是一串 token

而在Agentic 场景中：
CoT 是一段 tool-call 序列，即planning

在文本任务中：
用BLEU 或ROUGE等衡量生成文本与黄金答案在 token或字符层面的相似度，如具体的machine translation, summarization 等任务。

在Agentic 场景中
Process Accuracy，用来衡量实际 tool-call 序列与理想动作轨迹的匹配度，颗粒度（这个词挺合适的😅）是tool call 级别的。

如此一来，Kimi K2 在Agentic 场景中实现了与传统 NLP 的联系，使模型的规划与执行得以像文本生成那样可度量和优化。

[Tim Althoff @timalthoff]

Most complex tasks feature ill- and underdefined goals, at least initially.

Check out @vysrini's paper demonstrating that agents need to do a lot more than follow instructions, incl. refining goals, and balance competing objectives.

Current frontier models fail at these tasks

[Dongxi 东锡 NLP @dongxi_nlp]

Agentic 的文章，因为采用的post training recipe类似，方法同质化非常严重，容易带来审美疲劳，往往读完abstract 就不想往下读了。

但依然有一些文章，在RL天然task-specific的限制下，可以让task本身超越benchmark，颇具美感。

这几个月我个人最喜欢的几篇：

Absolute Zero：
高度地将代码类推理任务为三类： Deduction 演绎， Abduction 溯因， Induction 归纳。
因而，模型学习到的是这三类高度的任务，超越coding。

Test-Time Interaction：
关注的是Agent 决策中的两个基本核心策略：Exploration (探索) 与 Exploitation (利用)。

CURE：
利用不同Agent之间的关系，生成-验证，对抗-互测，完成自动监督，获得更加多样和动态的信号。

WebThinker：
不是让LLM 仅仅学会几个动作，而是让 LLM 真正掌握一套专业的 workflow。

RL的task-specific属性，让很多工作局限在数据集中，但恰恰是这个task，可以很具体，也可以实现高度的抽象。

就像Pretraining task，无论是masked language modeling还是next token prediction，非常简单直接，却能够让LLM发展出��决复杂问题的能力，大道至简。

[Nikhil Prakash @nikhil07prakash]

How do language models track mental states of each character in a story, often referred to as Theory of Mind?

Our recent work takes a step in demystifing it by reverse engineering how Llama-3-70B-Instruct solves a simple belief tracking task, and surprisingly found that it relies heavily on concepts similar to pointer variables in C programming!

[Andrew Piper @_akpiper]

Amazing thread. So much to unpack but for starters: “After writing, only 17 % of ChatGPT users could quote their own sentences, versus 89 % in the brain-only group.”

[Isabel Papadimitriou @isabelpapad]

Check out our ACL paper! We use shapley interactions to see which words (and phones) interact non-linearly -- what we lose when we assume linear relationships between features. Chat to Diganta in Vienna!

[Andrew Piper @_akpiper]

"Fiction-brained". This needs to be the next big cultural analytics project.

[Omar Shaikh @oshaikh13]

What if LLMs could learn your habits and preferences well enough (across any context!) to anticipate your needs?

In a new paper, we present the General User Model (GUM): a model of you built from just your everyday computer use.

🧵

[Lindia Tjuatja @lltjuatja]

When it comes to text prediction, where does one LM outperform another? If you've ever worked on LM evals, you know this question is a lot more complex than it seems. In our new #acl2025 paper, we developed a method to find fine-grained differences between LMs:

🧵1/9

[Jocelyn Shen @jocelynjshen]

Happy to share our preprint "Words Like Knives: Backstory-Personalized Modeling and Detection of Violent Communication" 🧵(1/9)

📄Paper: https://t.co/C4zJ5kvA3s

[Myra Cheng @chengmyra1]

Do people actually like human-like LLMs? In our #ACL2025 paper HumT DumT, we find a kind of uncanny valley effect: users dislike LLM outputs that are *too human-like*. We thus develop methods to reduce human-likeness without sacrificing performance.

[Andrew Piper @_akpiper]

In case you're curious whether you can use LLMs (including small local ones) for narrative topic modeling. The answer is yes! LLMs perform on par or better than humans at the task.

[Diyi Yang @Diyi_Yang]

@ysu_nlp second! also to "human-AI interaction" 😃

[Kabir @kabirahuja004]

📢 New Paper!

Tired 😴 of reasoning benchmarks full of math & code? In our work we consider the problem of reasoning for plot holes in stories -- inconsistencies in a storyline that break the internal logic or rules of a story��s world 🌎

W/ @melaniesclar, and @tsvetshop

1/n