Ruochen Zhao

NYU Math 2019 - Bachelor's
Harvard Data Science 2020 - Master's
NTU-NLP Current Ph.D. student!

[Wenxuan Zhang @Wenxuan__Zhang]

🤖 DR-Arena: an Automated Evaluation Framework for Deep Research Agents

Q: How do we reliably evaluate deep research agents that perform autonomous multi-step research on the live web?
A: Use Agents to evaluate Agents!

Deep research (DR) agents are rapidly proliferating, but evaluating them remains a major challenge. Because these agents operate in open, evolving environments, static benchmarks struggle with task generality, temporal misalignment, and data contamination.

We introduce DR-Arena, a fully automated, dynamic evaluation framework that:
1️⃣ Builds real-time Information Trees from live web sources
2️⃣ Tests both deep multi-hop reasoning and wide information coverage
3️⃣ Uses an adaptive evolvement loop to push agents to their capability boundaries

Notably, DR-Arena achieves a Spearman correlation of 0.94 with the Search Arena leaderboard, but without any human annotation, making it a scalable alternative to costly human evaluation.

🔗 Project page (paper, code, samples): https://t.co/9N06DF761l. Feedback welcome!

[Ruochen Zhao @RuochenZhao3]

Hi everyone!
This week, the WING.NUS Retrieval Augmented Generation lecture will cover important topics on Training: Fine Tuning, In Context Learning and Model Scaling!
Tune into the Week-05 Session | 6:00 PM SGT | Sep 11, 2025 https://t.co/TeuecBbpI2
@wing_nus & @knmnyn

[Victor.Kai Wang @VictorKaiWang1]

Customizing Your LLMs in seconds using prompts🥳!
Excited to share our latest work with @HPCAILab, @VITAGroupUT, @k_schuerholt, @YangYou1991, @mmbronstein, @damianborth : Drag-and-Drop LLMs(DnD). 2 features: tuning-free, comparable or even better than full-shot tuning.(🧵1/8)

[Jiachen ("Tianhao") Wang @JiachenWang97]

📰 Check out our new work on Data Shapley without retraining! Inspired by how privacy accountants track privacy leakage in DPSGD, we track data contributions by accumulating data value scores for each training *step*, in the spirit of divide-and-conquer.

[Ruochen Zhao @RuochenZhao3]

This is the advantage of Auto-Arena: completely automatic thus can keep up with everything in fashion! Chatbot arena hasn’t included Claude-3.5 yet, but we do! We can wait and see whether the results align.

[Wenxuan Zhang @Wenxuan__Zhang]

🚨 NEW MODEL RESULT ALERT 🚨

@AnthropicAI just released Claude 3.5 Sonnet, sparking a lot of buzz. But how does it perform?

We ran it through our Auto-Arena framework for evaluation — it outperforms GPT-4o and ranks at the top on the leaderboard (https://t.co/IP3Ao6CcUF) now🤯! In the direct comparison with GPT-4o with 40 questions, Claude 3.5 won 19, tied 4, and lost 17.

BTW, this underscores the power of such automated evaluation frameworks — we can get performance insights ahead of @lmsysorg's chatbot-arena rankings. Excited to see where it stands there too! 🤖

[Jinjie Ni @NiJinjie]

How to get ⚔️Chatbot Arena⚔️ model rankings with 2000× less time (5 minutes) and 5000× less cost ($0.6)?

Maybe simply mix the classic benchmarks.

🚀 Introducing MixEval, a new 🥇gold-standard🥇 LLM evaluation paradigm standing on the shoulder of giants (classic benchmarks).

https://t.co/igeaQuZASt

🕶️LLM Benchmark Mixture:
We mine comprehensive and well-distributed 🌎real-world user queries from the web and match them with similar queries from off-the-shelf 💯ground-truth-based benchmarks.

🤔Why to Use MixEval?
(1) 🎯 Accurate model ranking (0.96 correlation with Chatbot Arena)
(2) ⚡️ Fast, cheap, and reproducible execution, requiring only 6% the time and cost of MMLU
(3) 🌊 Dynamic benchmarking enabled by low-effort and stable updating mechanism
(4) 🏔️ Challenging question set (GPT-4o, the top model on MixEval leaderboard, achieves 64.7% accuracy)
(5) 🌌 Comprehensive and highly impartial query distribution, as it is deeply grounded in real-world user queries
(6) ⚖️ Fair grading process without preference bias, ensured by its ground-truth-based nature

❌ What's wrong with the current LLM evaluation?
(1)❓Query Bias: evaluation queries falling short of
comprehensiveness or appropriate distribution
a) ground-truth-based benchmarks
b) LLM-judged benchmarks
(2)👨‍⚖️Grading Bias: the grading process involving
significant bias or error
a) LLM-judged benchmarks
(3)🔬Generalization Bias: models overfitting the
evaluation data (contamination)
a) ground-truth-based benchmark
b) LLM-judged benchmarks

🤔 Any current benchmarks that are not so biased?
☑️ Yes. Large-scale user-facing benchmarks, e.g., ⚔️Chatbot Arena⚔️, solve
(1) query bias by collecting a large number of real-world user queries,
(2) grading bias by collecting a large number of real-world user preferences, and
(3) generalization bias by continuously receiving fresh queries.
But they are prohibitively 💰expensive (around $2936 for a single model, see the below figure), ⌚slow, and 🚫irreproducible!

✅MixEval addresses all these issues.
It's not only highly unbiased in query, grading, and generalization, but also fast, cheap, and reproducible.

📊We provide extensive meta-evaluation and insights for MixEval and existing LLM benchmarks in our paper. 🔥We hope this will deepen the community’s understanding of LLM evaluation and guide future research directions!

🏆Our dynamic leaderboard is now available at: https://t.co/VrkgE2DZRq

🚀Join us in revolutionizing LLM evaluation! Test your model on MixEval and see where you stand on our dynamic leaderboard.
🌊 We will update the data points on a monthly basis.

🚀 Moving forward, we'll continuously add new benchmarks to our pool as they release. This will refine our mixtures and enhance dynamism at a higher level.

This work is done by @NiJinjie, @XueFz, @xiangyue96, @yuntiandeng, Mahir Shah, Kabir Jain, @gneubig, and @YangYou1991. Kudos to the team!

We also sincerely thank @Francis_YAO_ @gblazex @zhansheng @_jasonwei @p_nawrot @soldni @guanzhi_wang @deepanwayx @BoLi68567011 @JunhaoZHANG19 @99Solaris @ZangweiZheng @zian_andy_zheng @KevinQHLin @WenhuChen @billyuchenlin and colleagues from NUS HPC-AI Lab & CMU NeuLab for insightful discussions and pointers!

[Wenxuan Zhang @Wenxuan__Zhang]

Want to keep up with the fast pace of LLM updates, but don’t want to wait a long time for Chatbot Arena @lmsysorg scores?
🎉🎉Check out our new project, Auto-Arena (https://t.co/82mHIHJtB9)!

TL;DR:
1. 🤖NO manual efforts required. Evaluation is now completely automated with LLM Agents.
We ask an 👨🏻‍🏫LLM examiner agent to draft questions, then two 🧑🏻‍🎓candidate agents to do a multi-round peer battle, and an 🧑🏻‍⚖️LLM judge committee to evaluate the outcomes.

2. 🧑🏻High alignment with human preferences!
Our method shows a SOTA 94.5% correlation with Chatbot Arena’s human scores, showing that it’s highly trustworthy and aligned with human preferences.

3. 💥Highly Flexible, you can evaluate any language / domain you want!
Auto-Arena can be easily adapted to other languages and domains, all you need to do is change the prompts! We tried Chinese as an example.

Everything is open-sourced:
Leaderboard: https://t.co/IP3Ao6CcUF
Website: https://t.co/8BVfNlg3BT
Blog: https://t.co/pP0Ggi6pqT
Paper: https://t.co/bgbvVtAMOe
Code: https://t.co/J7G0Sa5YMv

[Mathieu Ravaut @MatRavox]

Checkout our analysis paper, to be featured in ACL main conference:
https://t.co/k8H9AMN9s3
We investigate the "middle-curse" exposed by the LITM paper for the specific task of abstractive summarizarion. In brief: the middle curse is very much an issue ! w @JotyShafiq @AixinSG

[Caiming Xiong @CaimingXiong]

Are your LLMs highly accurate, or simply contaminated?
As the race to build the best LLM intensifies, clean evaluation is becoming more important than ever, yet contaminated LLMs and benchmarks obfuscate the real performance of models.
Checkout our new work (comprehensive survey + library) at NTU-NLP lab & Salesforce Research on the critical issue of contamination detection in LLMs, cc @ntunlp @MatRavox @D_Boss001 @HailinChen3 @XingxuanLi @RuochenZhao3 @FangkaiJiao @qcwntu @CaimingXiong @JotyShafiq
Paper:
https://t.co/16jXQ1hNXk
Library:
https://t.co/vNgXrto1XJ

[Li Xingxuan @XingxuanLi]

Chain-of-Knowledge📚 is accepted to #ICLR2024!

We present chain-of-knowledge (CoK), a novel framework that augments LLMs by dynamically incorporating grounding information from heterogeneous sources.

Paper link🔗: https://t.co/GMPT4Wgnr7
🧵(1/9)

[Ruochen Zhao @RuochenZhao3]

Our paper "Chain of Knowledge" is accepted to #ICLR2024 !

[Li Xingxuan @XingxuanLi]

We provide more analysis on:
(1) Single vs. multiple knowledge domains and sources
(2) Parallel vs. dynamic knowledge adapting
(3) Evaluating factuality improvement of the rationales

Please refer to our paper for details!
Paper link: https://t.co/GMPT4Wgnr7
🧵(9/9)

[ntunlpsg @ntunlpsg]

Check out our survey on open-source LLMs that catch up with #ChatGPT !

[M Saiful Bari (MARUF) @sbmaruf]

xCodeEval: An 🔥Execution-based 🔥Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation, and Retrieval
🏆 16.5B Tok
🏆 25M Doc
🥳 11 prog. lang.
⏬ Debunking memorization of ChatGPT💥
⏬ Reasoning Spectrum of LLMs 🧐
⏬ChatGPT analysis.👇

[M Saiful Bari (MARUF) @sbmaruf]

🔥A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets🔥
Yet another ChatGPT evaluation !!!! What's new ?
This time not automatic, human in a loop.
https://t.co/YeIs8cfiSa
Our ACL'23 paper covers FULL eval on benchmarks that actually MATTERS !!!

[Ruochen Zhao @RuochenZhao3]

Our work has been accepted to #ACL2023 !

[Ruochen Zhao @RuochenZhao3]

❌📖✅
How can we make LLMs output more correct answers? When humans are uncertain, we search for external references before answering. Now, we integrate exactly the same process into LLMs. Our Verify-and-Edit surpasses previous methods in QA tasks.
Paper: https://t.co/AR7UIlSta1

[Ruochen Zhao @RuochenZhao3]

With Palm-E and GPT4, Multimodal models are gaining popularity. To improve their grounding and correctness, retrieval augmentation is a promising solution... Check out our survey on methods that explore the intersection! https://t.co/NXU3aqSnHi

[Ruochen Zhao @RuochenZhao3]

Prof. Andrew Ng's tech blog has noted our discoveries, check it out here: https://t.co/kWigA9mGy8

[Ruochen Zhao @RuochenZhao3]

@lurker_tech @zacharynado (3/3) you’re welcome to verify if you’d like.” The first example he tried to pick a different one, but actually lacks factual grounding, as pointed out by the top comment. Overall, it has clear signs of copying and should have at least cited the sources when he published.

[Ruochen Zhao @RuochenZhao3]

@lurker_tech @zacharynado (2/3) intro and conclusion to be hasty summaries of overlapping content. 2. His examples are the same. The Gap example is exactly the same. For the Mexico nightclub example, he focused on reviews instead of opening times, which we mention as “the reviews are all different,