Adnan El Assadi

Research @Harvard,
Undergrad @Carleton_U,
AI & LLMs | MAEB, HUME

[Adnan El Assadi @AdnanElAssadi]

Did LLMs finally subsume embedding models?

We compared 10 LLMs vs. 26 embedding models across 37 MTEB tasks.

LLMs scored 77.6 while embedding models scored 77.2
But the costs are very different. 🧵

[Adnan El Assadi @AdnanElAssadi]

@Muennighoff @wregss I think an RLM @a1zhang would work here nicely

[Adnan El Assadi @AdnanElAssadi]

I think the context-window limitation is already well understood and implied, and also was explicitly mentioned in limitations: our corpora are 82–415 docs, which favours LLMs. The setup here is based on the corpus-in-context protocol (LOFT, Lee et al. 2024), not a claim that LLMs can replace retrieval for arbitrarily large corpora. The focus of the paper was on "embeddings" (with its various tasks) rather than search in its full generality. We already have upcoming work looking at agents for larger scale search and how they compare to embedding based retrieval.

[Adnan El Assadi @AdnanElAssadi]

@creon @Muennighoff multimodality is an extension!

[Stanford AI Lab @StanfordAILab]

The Embedder’s Dilemma— the ultimate question! Find out when you should use an embedding model vs an LLM!

[Orion Weller @orionweller]

If LLM price per intelligence continues to drop 10x a year, it’ll reach the $0.11 mark in … three ish years?

It’s beginning to look reasonable for people to use a LLM for tasks that used to be solely embedding-based (except for high latency or huge scale situations)

[Adnan El Assadi @AdnanElAssadi]

Excited to share that the paper was accepted to COLM 2026!

This was a great collaboration with @Muennighoff and @leejnhk.

Paper: https://t.co/XyI3ii6JLv
Code : https://t.co/WpfvuRi1pV

[Adnan El Assadi @AdnanElAssadi]

Did LLMs finally subsume embedding models?

We compared 10 LLMs vs. 26 embedding models across 37 MTEB tasks.

LLMs scored 77.6 while embedding models scored 77.2
But the costs are very different. 🧵

[Adnan El Assadi @AdnanElAssadi]

We call this the Embedder’s Dilemma.

LLMs now match the best embedding models but at much higher cost and lower throughput.

We study how this tradeoff changes across different task types.

[Adnan El Assadi @AdnanElAssadi]

Excited to see MVEB out 🎥

It’s a 23-task benchmark in MTEB covering retrieval, clustering, classification, and video-text QA, and really highlights how uneven current video embedding models still are across tasks.

[Niklas Muennighoff @Muennighoff]

we just released: MVEB: Massive Video Embedding Benchmark 🎥

with more ai-generated videos, good video embeddings may become key, as you cant just grep through videos like for text..

[Adnan El Assadi @AdnanElAssadi]

Grateful to have been part of this. SWE-Marathon pushes coding agents on long-horizon, full-stack work existing benchmarks don't really capture, and releasing the trajectories and rollout logs is the part I'm most glad we did. Worth a read: https://t.co/8EAma5SGL3

[Rishi Desai @rishi_desai2]

Can coding agents stay coherent over a 1 billion token budget?

Can they build Slack from scratch?
Rewrite a JAX codebase in PyTorch?
Build a C compiler in Rust?

Enter SWE-Marathon: a benchmark for autonomous long-horizon software work.

[Adnan El Assadi @AdnanElAssadi]

Kudos to @dianetc_ , @lateinteraction for creating the benchmark.
Paper: https://t.co/7liwLluY3v
Dataset: https://t.co/WizFgrjwRe

[Adnan El Assadi @AdnanElAssadi]

You can now run OBLIQ-Bench through MTEB.

OBLIQ-Bench tests retrieval on oblique queries, where relevance is latent and barely shows up on the surface, exactly the cases current retrieval paradigms miss.

Any MTEB-supported embedding model can now be evaluated on it directly.

[Niklas Muennighoff @Muennighoff]

We released MAEB: Massive Audio Embedding Benchmark🎵 mteb now covers audio/image/text embedding! See the leaderboard for the top audio embedding models🙂

LB: https://t.co/VwNN9zy2ff
Paper: https://t.co/IY0TwqeZzm