Connor Shorten

AI and Databases @weaviate_io

[Connor Shorten @CShorten30]

Hey everyone! I am super excited to share that our new research report is live on ArXiv! 🎉

IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering!

Thread with more details! 🧵(1/11)

[Connor Shorten @CShorten30]

@ChiefScientist *Ultrahigh effort

https://t.co/YTKuGZrMXa

[Weaviate Podcast @weaviatepodcast]

What happens if you skip first-stage retrieval and let a state-of-the-art cross-encoder score all 10,000 documents itself? 🤔

In Drowning in Documents, BM25 beat it.

In this chapter, @mat_jacob1002 walks through the full-scoring experiments: golds plus 10,000 randomly sampled documents, every one scored by the reranker, and old-school keyword search still came out ahead.

Watch Mathew break down the experiment 👇
https://t.co/yhvQMooe0m

[Connor Shorten @CShorten30]

@melissapan @mat_jacob1002 Thanks Melissa! 💚🚀

[Melissa Pan @melissapan]

Check out this podcast with @mat_jacob1002 if you are interested in reranking / retrieval related research!

Mat also shared a bit about our work on pushing the pareto frontier of retrieval agents 🚀

[Connor Shorten @CShorten30]

@aaxsh18 Awesome! Thanks Aamir! 💚

[Connor Shorten @CShorten30]

Scaling Test-Time Compute has been one of the clearest wins for improving AI systems. I am SUPER EXCITED to share that we’re bringing it to Search with new effort tiers in the Query Agent’s Search Mode!

Medium, high, and ultrahigh 🚀

🔬We benchmarked all three tiers against Hybrid Search across 8 reasoning-intensive and domain-specific retrieval benchmarks: 5 subsets from BRIGHT, IRPAPERS, WixQA, and OBLIQ-Bench Congress.

We find that every effort tier outperforms Hybrid Search, often by a wide margin. Ultrahigh effort roughly quadruples Hybrid Search’s nDCG@10 on BRIGHT Biology, and delivers a ~7x improvement in Success@1 on OBLIQ-Bench Congress. More details can be found in the image below!

I hope you find this interesting!

YouTube: https://t.co/NZTSrxO0xr
Blog: https://t.co/Zlm45ZvBQS
Docs: https://t.co/NJfpEpCGzw

[Weaviate Podcast @weaviatepodcast]

Retrieve more documents, rerank them, get better results. That's everyone's mental model of search. But push a cross-encoder past around 100 documents and recall@10 starts plummeting instead. 🤔

In this chapter, @mat_jacob1002 traces how Drowning in Documents started: a drop so counterintuitive it looked like a bug, and the realization that rerankers act less like stronger retrievers and more like boosting, fitting the first stage's errors. 👇

https://t.co/ZJ0wSnSE4C

[Connor Shorten @CShorten30]

Retrieve more, rerank, get better results. That's what most of us expect from search, but it turns out to be wrong! 🤯

I'm SUPER EXCITED to publish the 141st episode of the Weaviate Podcast with Mathew Jacob (@mat_jacob1002)! Mathew led the work behind "Drowning in Documents" during his time at Databricks and is now a Ph.D. student at the University of Washington working on ML systems!

This episode dives deep into "Drowning in Documents". This has been one of the most influential papers for us @weaviate_io as we are exploring scaling reranked retrieval. I think it is a must read for those working in Search and Information Retrieval. 🙌

We begin with an overview of the paper, and then dive into full scoring with cross encoders and phantom hits. We then cover Listwise Rerankers, what next generation cross encoders might look like, and ranking cascades.

On the topic of ranking cascades, I loved learning about Mathew’s work with Melissa Pan (@melissapan), Negar Arabzadeh (@NegarEmpr) and collaborators on “Natural Language Query to Configuration for Retrieval Agents”. Per-query “effort” prediction is certainly going to be a huge component on the future of these search systems! (Congratulations to OpenRouter 😆)

We then dove into Mathew’s work on TraceLab, a super exciting effort to understand coding agents such as Claude Code and Codex. ⌨️

This was a super fun conversation, and I really hope you find it useful!
YouTube: https://t.co/3lPicYhk5R
Spotify: https://t.co/sx0o06kilI

[Connor Shorten @CShorten30]

@aaxsh18 Would love to! 🔥

[Connor Shorten @CShorten30]

@ConorBronsdon @mat_jacob1002 Thanks Conor! 💚 @bobvanluijt is on the record for a trophy at 150 🏆😆

[Connor Shorten @CShorten30]

Retrieve more, rerank, get better results. That's what most of us expect from search, but it turns out to be wrong! 🤯

I'm SUPER EXCITED to publish the 141st episode of the Weaviate Podcast with Mathew Jacob (@mat_jacob1002)! Mathew led the work behind "Drowning in Documents" during his time at Databricks and is now a Ph.D. student at the University of Washington working on ML systems!

This episode dives deep into "Drowning in Documents". This has been one of the most influential papers for us @weaviate_io as we are exploring scaling reranked retrieval. I think it is a must read for those working in Search and Information Retrieval. 🙌

We begin with an overview of the paper, and then dive into full scoring with cross encoders and phantom hits. We then cover Listwise Rerankers, what next generation cross encoders might look like, and ranking cascades.

On the topic of ranking cascades, I loved learning about Mathew’s work with Melissa Pan (@melissapan), Negar Arabzadeh (@NegarEmpr) and collaborators on “Natural Language Query to Configuration for Retrieval Agents”. Per-query “effort” prediction is certainly going to be a huge component on the future of these search systems! (Congratulations to OpenRouter 😆)

We then dove into Mathew’s work on TraceLab, a super exciting effort to understand coding agents such as Claude Code and Codex. ⌨️

This was a super fun conversation, and I really hope you find it useful!
YouTube: https://t.co/3lPicYhk5R
Spotify: https://t.co/sx0o06kilI

[Weaviate Podcast @weaviatepodcast]

Weaviate Podcast #141 is live! 🎙️💚🎉

Drowning in Documents with @mat_jacob1002!

[Mathew Jacob @mat_jacob1002]

Finally, have to shoutout @KanZhu854772 for his amazing work with TraceLab. I would stay tuned for some of the work he's going to drop soon, going to be 🔥.
Should also follow @UWSyFi while you're here to keep up with the latest from our group at UW!

[Connor Shorten @CShorten30]

@mat_jacob1002 💚 Thank you so much Mathew! Learned so much from the conversation!

Yes! Super excited to build on this work and scale reranked retrieval! 🚀

[Mathew Jacob @mat_jacob1002]

Had so much fun talking to @CShorten30 about our Drowning in Documents work! Was fun going down memory lane about this work :D! Stay tuned for the research that is going to be coming out of Weaviate👀

[Connor Shorten @CShorten30]

@mat_jacob1002 @mrdrozdov @lateinteraction @matei_zaharia @mcarbin @erikmlindgren What a team! 🔥💚

[Mathew Jacob @mat_jacob1002]

Also shoutout to my amazing coauthors @mrdrozdov @lateinteraction @matei_zaharia @mcarbin @erikmlindgren for their amazing mentorship during the Drowning in Documents project and helping me push this project to completion!

[Connor Shorten @CShorten30]

@zainhas Did someone say.... effort ?! 😆

https://t.co/J0YCNSH3X6

[Connor Shorten @CShorten30]

@ChaitanyaK57 @GoogleAIStudio 🔥 super interesting to see web search > reranker - would be cool to also mix results from Weaviate :)