LLM4Eval Workshop

The Third Workshop on Large Language Models for Evaluation in IR (LLM4IREval@SIGIR2025)

[LLM4Eval Workshop @llm4eval]

LLMs are reshaping how we evaluate ranked lists, conversations, and even simulate users.

At #LLM4Eval @ #SIGIR2025, we’re spotlighting 3 special themes:
🧠 Beyond query-doc
🗣️ Simulating users
🧪 Synthesizing evaluation data
👇 Curious? Read on & submit by April 23!
#LLMs #IR

[LLM4Eval Workshop @llm4eval]

📢 Call for Papers: LLM4EVAL @ SIGIR '25 - The 3rd Workshop on Large Language Models for Evaluation in IR! @SIGIRConf

🗓️ Submission: April 23, 2025 (AoE)
🔗 Full CFP: https://t.co/B88X3U5wWX
🌐 Workshop: https://t.co/mc84kebwB0

[LLM4Eval Workshop @llm4eval]

Chirag Shah is this morning giving a keynote on: From Relevance to Reality: Scaling Human-Centered Evaluation in the LLM Era

[LLM4Eval Workshop @llm4eval]

You are welcome to join us at Giotto Platea for the LLM4Eval workshop #SIGIR2025 .

[LLM4Eval Workshop @llm4eval]

The deadline for submitting your LLM evaluation paper is tomorrow. Looking forward to interesting contributions!

[LLM4Eval Workshop @llm4eval]

📢 Deadline extended!
Submit to #LLM4Eval @ #SIGIR2025 by April 30 (AoE)

We welcome work on LLM-based evaluation, including:
🧠 beyond query-doc
🗣️ simulating users
🧪 synthetic eval data

🔗 https://t.co/B88X3U5wWX

#CFP #IR #LLMs #SIGIR2025

[LLM4Eval Workshop @llm4eval]

cc @cnsiro @srahmanidashti @maliannejadi @nickcr @claclarke @guglielm0f @UnderdogGeek @pt_ir @emine_yilm

[LLM4Eval Workshop @llm4eval]

📢 Deadline extended!
Submit to #LLM4Eval @ #SIGIR2025 by April 30 (AoE)

We welcome work on LLM-based evaluation, including:
🧠 beyond query-doc
🗣️ simulating users
🧪 synthetic eval data

🔗 https://t.co/B88X3U5wWX

#CFP #IR #LLMs #SIGIR2025

[LLM4Eval Workshop @llm4eval]

LLMs are reshaping how we evaluate ranked lists, conversations, and even simulate users.

At #LLM4Eval @ #SIGIR2025, we’re spotlighting 3 special themes:
🧠 Beyond query-doc
🗣️ Simulating users
🧪 Synthesizing evaluation data
👇 Curious? Read on & submit by April 23!
#LLMs #IR

[Mohammad Aliannejadi @maliannejadi]

We will host multiple breakout discussions to pick your brains 🧠 on the future challenges and research on this topic. Let's shape the LLMEval research agenda.

Interested? Already have thoughts and ideas? Then, submit your extended abstracts or just join us in the workshop.

[LLM4Eval Workshop @llm4eval]

🧵 Theme 3:

🧪 Synthesizing evaluation data: When judgments are scarce, can we generate evaluation data with LLMs?

We want approaches for creating evaluation datasets in low-resource or privacy-sensitive scenarios, where collecting real data is hard, expensive, or impossible.

[LLM4Eval Workshop @llm4eval]

LLMs are reshaping how we evaluate ranked lists, conversations, and even simulate users.

At #LLM4Eval @ #SIGIR2025, we’re spotlighting 3 special themes:
🧠 Beyond query-doc
🗣️ Simulating users
🧪 Synthesizing evaluation data
👇 Curious? Read on & submit by April 23!
#LLMs #IR

[LLM4Eval Workshop @llm4eval]

📢 Call for Papers: LLM4EVAL @ SIGIR '25 - The 3rd Workshop on Large Language Models for Evaluation in IR! @SIGIRConf

🗓️ Submission: April 23, 2025 (AoE)
🔗 Full CFP: https://t.co/B88X3U5wWX
🌐 Workshop: https://t.co/mc84kebwB0

[LLM4Eval Workshop @llm4eval]

🧵 Theme 2:

🗣️ Simulating users: What if LLMs could act as realistic user proxies?

We are looking for papers on simulating user behavior, preferences, context, or personas for evaluation.

[LLM4Eval Workshop @llm4eval]

Organized by @cnsiro, @srahmanidashti , @maliannejadi, @nickcr , @claclarke, @guglielm0f , @UnderdogGeek , @pt_ir , and @emine_yilm

#SIGIR, #IR, #Evaluation, #LLM4Eval

[LLM4Eval Workshop @llm4eval]

📢 Call for Papers: LLM4EVAL @ SIGIR '25 - The 3rd Workshop on Large Language Models for Evaluation in IR! @SIGIRConf

🗓️ Submission: April 23, 2025 (AoE)
🔗 Full CFP: https://t.co/B88X3U5wWX
🌐 Workshop: https://t.co/mc84kebwB0

[LLM4Eval Workshop @llm4eval]

In this series, we move BEYOND search relevance to tackle:
✨ Personalized search evaluation
✨ Conversational interfaces
✨ Holistic assessment of integrated experiences
✨ Bias & fairness in LLM evaluations

[Tech At Bloomberg @TechAtBloomberg]

In today's 2nd Workshop on #LLMs for Evaluation in #InformationRetrieval (#LLM4Eval) at #WSDM2025 (09:15 CET), @edgarmeij, our Head of #AI Platforms Engineering, will deliver a keynote on "Synthetic Evaluations & #GenAI Application Development"
https://t.co/CcX7AV1rFy
#AI

[Hossein A. (Saeed) Rahmani @srahmanidashti]

The workshop will focus on models, techniques, data collections, and methodologies for information retrieval evaluation in the era of LLMs.

#WSDM2025 #IR #Evaluation #Performance #Workshop #NLP #SIGIR

[Hossein A. (Saeed) Rahmani @srahmanidashti]

Organiser: @srahmanidashti, @cnsiro, @maliannejadi, @nickcr, @claclarke, @guglielm0f, @UnderdogGeek, @pt_ir , and @emine_yilm

[Hossein A. (Saeed) Rahmani @srahmanidashti]

The call for papers is now open for the second workshop of #LLM4Eval:

💡 See CFP: https://t.co/IdJKD19Vt1
🗓 Submission deadline: January 20, 2025

[Hossein A. (Saeed) Rahmani @srahmanidashti]

🔊 The second workshop on Large Language Models for Evaluation in IR (#LLM4Eval), co-located with #WSDM2025, is now accepting submissions! If your work explores or leverages LLM capabilities for evaluation, we encourage you to submit and share your research.

[Hossein A. (Saeed) Rahmani @srahmanidashti]

📢 Thrilled to announce that, "SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval" is publicly available.

w/ @wangxieric, @emine_yilm, @nick_craswell, @UnderdogGeek, @pt_ir

Check it out: https://t.co/Omqt9M7z0x

[Guglielmo Faggioli @guglielm0f]

Great panel discussion with @ian_soboroff, @lauradietz99, @claclarke, Michael Ekstrand moderated by @UnderdogGeek at @llm4eval. #SIGIR24