RAG and in-context learning are the go-to approaches for integrating new knowledge into LLMs, making inference very inefficient
We propose instead 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗠𝗼𝗱𝘂𝗹𝗲𝘀 : lightweight LoRA modules trained offline that can match RAG performance without the drawbacks
Automating Multi-Agent Design:
🧩Multi-agent systems aren’t just about throwing more LLM agents together.
🛠️They require mastering the subtle art of prompting and agent orchestration.
Introducing MASS🚀- Our new agent optimization framework for better prompts and topologies!
We achieved the first instance of successful subword-to-byte distillation in our (just updated) paper.
This enables creating byte-level models at a fraction of the cost of what was needed previously.
As a proof-of-concept, we created byte-level Gemma2 and Llama3 models.
🧵
🚀Let’s Think Only with Images.
No language and No verbal thought.🤔
Let’s think through a sequence of images💭, like how humans picture steps in their minds🎨.
We propose Visual Planning, a novel reasoning paradigm that enables models to reason purely through images.
We created Approximate Likelihood Matching, a principled (and very effective) method for *cross-tokenizer distillation*!
With ALM, you can create ensembles of models from different families, convert existing subword-level models to byte-level and a bunch more🧵
I am hiring a Student Researcher for our Modularity team at the Google DeepMind office in Zurich🇨🇭
Please fill out the interest form if you would like to work with us! The role would start mid/end 2025 and would be in-person in Zurich with 80-100% at GDM
https://t.co/Vfypj91KHy
📣Happy to (pre-)release my Fleurs-SLU benchmark to evaluate massively multilingual spoken language understanding on SIB & Belebele. Work done at @Mila_Quebec with @davlanade@gg42554@licwu
Datasets:
https://t.co/wqSfkT3VA3
https://t.co/882nh8znY1
Details to follow👇
Thrilled to share our updated paper: "UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models"
We propose a new robust LLM unlearning method via Self-Distillation on Adjusted Logits (UNDIAL).
📄 Paper: https://t.co/vqX1YuFF5e
Do you DARE?
Introducing a multiple-choice VQA benchmark ✨DARE✨ with:
- 4 main robustness evaluation ⛓️
- 5 diverse categories 🧩
- Extensive analysis of 4 widely used VLMS 🤖
As someone who spent years working in multilingual NLP, I am so happy that we're finally seeing (L)LMs and (N)MT systems working in tandem towards the shared cause. The idea in this work is so simple & sweet, and yet it moves! 🌍🌏🌎
Introducing NLLB-LLM2Vec! 🚀
We fuse the NLLB encoder & Llama 3 8B trained w/ LLM2Vec to create NLLB-LLM2Vec which supports cross-lingual NLU in 200+ languages🔥
Joint work w/ Philipp Borchert, @licwu, and @gg42554 during my great research stay at @cambridgeltl
Which output is better?
[A] or [B]? LLM🤖: B❌
[B] or [A]? LLM🤖: A✅
Thrilled to share our preprint in addressing preference biases in LLM judgments!🧑⚖️We introduce ZEPO, a 0-shot prompt optimizer that enhances your LLM evaluators via fairness⚖️
📰Paper: https://t.co/ZkMvJnFFMC
Excited to introduce TopViewRS: VLMs as Top-View Spatial Reasoners🤖
TopViewRS assess VLMs’ spatial reasoning in top-view scenarios🏠just like how you read maps🗺️
Spoiler🫢GPT4V and Gemini are neck-and-neck, each excelling in different setups but neither even close to us humans
Introducing Zero-Shot Tokenizer Transfer (ZeTT) ⚡
ZeTT frees language models from their tokenizer, allowing you to use any model with any tokenizer, with little or no extra training.
Super excited to (finally!) share the first project of my PhD🧵
Adapters are just a great way to share/benefit from new capabilities without handing around the kitchen sink.
Congrats to the AdapterHub folks for adding support for quantized training (Q-LoRA and friends).
If we align LLMs through preferences, perhaps we should also evaluate them the same way (and respect transitivity)? The answer is: yes, we should. The trick, however, is how to make evaluation tractable. If you are into the whole "LLM-as-Judges" line of work, check this paper!
🔥New paper!📜
Struggle to align LLM evaluators with human judgements?🤔
Introducing PairS🌟: By exploiting transitivity, we push the potential of pairwise preference in efficient ranking evaluations that has better alignment!🧑⚖️
📖https://t.co/W4wSHQqdYc
💻https://t.co/q5ZMGkvaaj
I am still looking for PhD students starting in September 2024! The deadline to apply for the CDT in NLP is the 11th of March.
If you wish to do research in modular and efficient LLMs, here are some highlights of my lab's research from the past year ⬇️🧵
Think globally, act locally? Well, we were thought-experimenting whether LLMs would understand people from different places around our hometowns better than we ever might... And then we have eventually decided to make an actual (non-thought) experiment out of these thoughts! 👇👇
Interested in commonsense reasoning in dialectal texts? The DIALECT-COPA shared task is the perfect fit for you, providing train and dev data for four official South-Slavic languages and two out of three related test dialects https://t.co/im2CzBFZjY @vardialworkshop@naaclmeeting
We scaled sparse fine-tuning (SFT) to LLMs (such as Llama 2) by making it both parameter- and memory-efficient!
(q)SFT instruction tuning performance is often better than (q)LoRA with comparable speed and memory load.
Paper: https://t.co/wGew8XQvdW
Code:
https://t.co/zElZ7BCbJ6 (SFT PEFT) https://t.co/sOB4WVOHm5 (experiments)
@AlanAnsell5@licwu@h_sterz@annalkorhonen