On a career break
Prev. Director of AI and Staff Research Engineer @Hyperbots_Inc
IIT Bombay 5th Year CS PhD @cfiltnlp @iitbombay
Building in stealth 🚀
🇮🇳Releasing resources for Multilingual Search in 11 Indian languages!
1⃣INDIC-MARCO (Translated version of MSMARCO in 11 Indian Languages): https://t.co/R7K69S0cSR
2⃣Indic-ColBERT (11 Multilingual ColBERT Models): https://t.co/XE4YoMZYOt
Paper: https://t.co/a9Rl3Jo87w
overwhelming evidence for late interaction / multi-vector models yet again :-)
> even after finetuning, single-vector models lag far behind multi-vector embeddings, which achieve significant performance gains and exhibit greater robustness to catastrophic forgetting.
The “grep-is-all-you-need” nonsense arguments arise from the fact that too many people think neural search means single-vector IR, which do in fact suck. But we’ve known that since 2019.
Quoting @aaxsh18, CEO of Mixedbread:
> late interaction cant stop winning
Of course not. In fact, my lab is simultaneously building RLMs as the next paradigm for LLMs *and* developing the next paradigm for retrieval (stay tuned!).
Retrieval will not go anywhere: if you have a large corpus with, say, billions of tokens over which you issue many queries, you necessarily need to build some index data structures that enable fast sub-linear access.
RLMs may internally choose to build such an index when it proves to be an effective tool, but fundamentally RLMs are about long one-off context. You wouldn’t typically put an RLM over a million documents and expect that to be the optimal system design.
(Thank you for the question @jayitabhattac11 !)
For those interested in making OSS contributions to the RLM repo, I've added a bunch of random thoughts and TODOs of what to add in a *messy* Markdown file on the GH repo.
Feel free to tackle any of them, or any other things you think are meaningful. I'll be pretty active here or on the repo. Once I finish some other related work, I might open up a Discord channel or something for people who want to make longer standing contributions to the repo / discuss the direction of where to take it. Cheers!
https://t.co/EVK7g0vzf0
@a1zhang IMO, RLMs are as “language model”-y as modern “LLMs” or Reasoning Models are truly “statistical models of language”.
All three are a bit of a stretch BUT in the same way.
Pedantically, all three are language processing systems, eg recursive/reasoning language processing system.
> You’ll implement ColBERT to understand multi-vector search [and] apply ColPali for patch-level image retrieval.
So happy to see the great folks at @DeepLearningAI@AndrewYNg host a course on late interaction (ColBERT, ColPali et al) after their short course on DSPy :D
Please consider applying to the program. Over two years, my research skills, perspective on research have all been broadened and sharpened. This is an exceptional group, in the way they groom you, and allow you a room for exploring wild ideas. Pls reach out if you have questions!
Martin @martin_casado and I had a fun hour-long chat about why we need an AI software layer, and why that's true even if AGI arrives.
This is basically my take on why "the model" is definitely NOT "the product", though models are one way you may decide to implement some products
Happy Friday Everyone,
DSPyWeekly Issue #11 is live! 🚀
Highlights:
🔹 A cookbook for Self-Evolving Agents
🔹 Teaching local models tool-calling
🔹 New DSPy + Neo4j integration
🔹 A new "Events" section to track DSPy meetups!
Plus new projects like codex_dspy & AUTODSPy.
#DSPy #AI #LLMs #AgenticAI #Neo4j
@lateinteraction PEFT as an idea is clean and modular. LoRA is a bit of a hack that happens to work. Yet experimentally, it is the most effective PEFT.
The labs don't want you to know this (jk) but they have no clue how to best prompt their own models either. To some approximation, you just pre-/post-train it on a lot of data, intervene on certain behaviors, and what comes out is what comes out.