Nour Jedidi

CS PhD @UWaterloo | Interests: IR & NLP

[Nour Jedidi @nour_jedidi]

To make an LLM a strong dense retriever, you often need training…

But what if you only have a few-examples 🥩🥦 and no training budget?

Just add RICE 🍚, a simple way to turn your LLM into an effective dense retriever that requires zero training 👨‍🍳

📄 https://t.co/A1dQGj4ShG

[Nour Jedidi @nour_jedidi]

The official version of our paper “Don’t ‘Overthink’ Pointwise Reranking: Is Reasoning Truly Necessary?” is out! 🎉

📄 https://t.co/wPioDrMACN

See the 🧵 below for details! 👇

[Nour Jedidi @nour_jedidi]

Excited to share that our paper was accepted to EMNLP Findings 🎉

We ask a simple question: Does reasoning improve pointwise rerankers?

The answer? It can actually hurt... even for Rank1, a reranker trained to reason-then-answer 😱

📄 https://t.co/ZyL7fYghDL

[Jimmy Lin @lintool]

tl;dr - the team @Sahel_Sharify @Lingwei_Gu @nour_jedidi speedran nanochat by @karpathy using ClimbMix by @shizhediao et al. and then SFT'ed on relevance data from @beirmug @crystina_z to arrive at a competitive pointwise reranker: https://t.co/QQgZ2rk4my

[Jimmy Lin @lintool]

What are we up to? 🏡 Project Greenhouse: Progress Toward Fully Open and Sovereign Agentic Search w/ @Sahel_Sharify @Lingwei_Gu @nour_jedidi https://t.co/QQgZ2rjwx0

[Nour Jedidi @nour_jedidi]

@sivareddyg @sivareddyg thanks for the kind words!

This is an interesting idea! In-context examples could further refine the output-centric embeddings by steering it toward the kinds of outputs we want for retrieval - this may even work out of the box since LLM2Vec-Gen keeps LLM frozen too

[Nour Jedidi @nour_jedidi]

To make an LLM a strong dense retriever, you often need training…

But what if you only have a few-examples 🥩🥦 and no training budget?

Just add RICE 🍚, a simple way to turn your LLM into an effective dense retriever that requires zero training 👨‍🍳

📄 https://t.co/A1dQGj4ShG

[Nour Jedidi @nour_jedidi]

This work was done with @YungSungChuang, Jim Glass and @lintool!

[Nour Jedidi @nour_jedidi]

Excited to share that our paper was accepted to EMNLP Findings 🎉

We ask a simple question: Does reasoning improve pointwise rerankers?

The answer? It can actually hurt... even for Rank1, a reranker trained to reason-then-answer 😱

📄 https://t.co/ZyL7fYghDL

[Yung-Sung Chuang @YungSungChuang]

🚨Do passage rerankers really need explicit reasoning?🤔—Maybe Not!

Our findings:
⚖️Standard rerankers outperform those w/ step-by-step reasoning!
🚫Disable reasoning from reasoning reranker actually improves reranking accuracy!🤯
👇But, why?

📰https://t.co/vg5XwoxIPd

(1/6)

[Nour Jedidi @nour_jedidi]

So, can reasoning still help pointwise rerankers? 🧐

We tried self-consistency, inverse training (answer-then-reason), and graded relevance labels, but none outperformed Rank0...

Our takeaway? For pointwise reranking, reasoning doesn’t give enough bang for the buck 💸

[Nour Jedidi @nour_jedidi]

@CShorten30 Thank you, @CShorten30!

[Nour Jedidi @nour_jedidi]

So, if you want dense embeddings and have an LLM but no training budget…don’t forget to add RICE 🍚👨‍🍳

This work was done with @abdbasitali, @hanglics, and @lintool!

[Nour Jedidi @nour_jedidi]

👉 We can also spend inference-time compute on choosing better examples 💡

Selecting query-specific demonstrations improves effectiveness 📈

[Nour Jedidi @nour_jedidi]

👉 Interestingly, correct query–document pairings isn’t the key ingredient 🤔

In our ablations, non-relevant and random pairs from the same corpus outperform relevant cross-corpus examples, suggesting the domain language matters more than the accuracy of the examples…

[Nour Jedidi @nour_jedidi]

👉 Versus other dense retrieval methods like HyDE which work “out-of-the-box”, RICE 🍚 is the most effective

👉 Compared to LLM2Vec-Gen - an MTEB SOTA for self-supervised embeddings - RICE 🍚 remains competitive despite relying on *zero* parameter updates

[Nour Jedidi @nour_jedidi]

The RICE recipe 👨‍🍳

1️⃣ Start with your few query–document pairs 🥩🥦
2️⃣ Use PromptReps to generate one representative word for each example document
3️⃣ Use these pairs and words as demonstrations when encoding both queries and documents

Here’s what we find 👇

[Nour Jedidi @nour_jedidi]

This is what motivates our core question: Can we prompt LLMs, using a few human-labeled examples, to build effective representations for dense retrieval?

Our answer: Yes 🔥

The missing ingredient? RICE 🍚

[Nour Jedidi @nour_jedidi]

However, 1️⃣ still requires encoder training, which our training budget doesn’t allow 😔 and 2️⃣ leaves our examples 🥩🥦 unused!

[Nour Jedidi @nour_jedidi]

First, how can we build dense retrievers with limited labeled data? 💡

1️⃣ Use your few examples to prompt an LLM to generate task-specific training data 🐊

Or you can just skip training ...

2️⃣ Prompt the LLM to produce retrieval representations directly, PromptReps-style 💬