Maor Ivgi

NLP researcher / Ph.D. candidate at Tel-Aviv University

[Maor Ivgi @maorivg]

1/5 🧠 Excited to share our latest paper focusing on the heart of LLM training: data curation! We train a 7B LLM achieving 64% on 5-shot MMLU, using only 2.6T tokens. The key to this performance? Exceptional data curation. #LLM #DataCuration

[Vaishaal Shankar @Vaishaal]

I am really excited to introduce DataComp for Language Models (DCLM), our new testbed for controlled dataset experiments aimed at improving language models. 1/x

[Amanda Bertsch @abertsch72]

Super honored to win the Language Modeling SAC award!

I'll be presenting this work Wednesday in the 2pm poster session in Hall 3-- would love to chat with folks there or at the rest of the conference about long context data, ICL, inference time methods, New Mexican food, etc :)

[Or Patashnik @OPatashnik]

Ever wondered how a SINGLE token represents all subject regions in personalization? Many methods use this token in cross-attention, meaning all semantic parts share the same single attention value. We present Nested Attention, a mechanism that generates localized attention values

[Mor Geva @megamor2]

What's in an attention head? 🤯

We present an efficient framework – MAPS – for inferring the functionality of attention heads in LLMs ✨directly from their parameters✨

A new preprint with @AmitElhelo 🧵 (1/10)

[Alex Dimakis @AlexGDimakis]

https://t.co/OBsYc5udSr
I’m excited to introduce Evalchemy 🧪, a unified platform for evaluating LLMs. If you want to evaluate an LLM, you may want to run popular benchmarks on your model, like MTBench, WildBench, RepoBench, IFEval, AlpacaEval etc as well as standard pre-training metrics like MMLU. This requires you to download and install more than 10 repos, each with different dependencies and issues. This is, as you might expect, an actual nightmare. (1/n)

[Leshem (Legend) Choshen 🤖🤗 @ACL @ICML @LChoshen]

Scaling laws predict🦣large models by training🦟small ones, cool right?
Fortunately, they are not that complicated or costly
at least they don't have to be

We have collected 400+ models
fitted 1000+ scaling laws
and created 1 guide
for cheap & more reliable scaling law fitting:

[Talor Abramovich @AbramovichTalor]

We're launching EnIGMA, our state-of-the-art AI agent for offensive cybersec!
It uses tools like Ghidra & pwntools, can debug, connect to servers, and exploit vulnerabilities to solve CTF challenges.
Built with researchers from Princeton, NYU, and TAU.
https://t.co/AmNYz89kcO

[Ofir Press @OfirPress]

We just gave SWE-agent offensive cybersecurity capabilities, leading to state-of-the-art results on two challenging benchmarks!

Try it out, it's live now

[Luca Soldaini 🎀 @soldni]

Olmo goes multimodal!

We are launching Molmo, a open family of multimodal models that rival the best closed VLMs out there 🤯

We spent the last 9 months meticulously curating PixMo, a dataset of (a) high-quality image-caption pairs and (b) multimodal instruction data.

[Ben Bogin @ben_bogin]

📢 New Benchmark: SUPER for Setting UP and Executing tasks from Research repositories

Reproducibility is crucial in science. We introduce SUPER to evaluate LLMs' capabilities in autonomously running experiments from research repositories. ⬇️

https://t.co/U47r3F3UO5

[Theophile Gervet @theo_gervet]

Excited to finally share what I have been working on at @MistralAI. Meet Pixtral 12B, our first-ever multimodal model:
- Drop-in replacement for Mistral Nemo 12B
- SOTA multimodal capabilities without compromising on SOTA text-only capabilities
- New 400M parameter vision encoder trained from scratch, paired with a 12B multimodal decoder based on Mistral Nemo
- Handles variable image sizes and aspect ratios
- Supports multiple images in 128k context window
- Apache 2.0 license

Blog: https://t.co/2pbjn6lPDd
Weights: https://t.co/CidiN2Emtu
Try it: https://t.co/p4IuFcEsSb

1/N

[Niklas Muennighoff @Muennighoff]

Releasing OLMoE - the first good Mixture-of-Experts LLM that's 100% open-source
- 1B active, 7B total params for 5T tokens
- Best small LLM & matches more costly ones like Gemma, Llama
- Open Model/Data/Code/Logs + lots of analysis & experiments

📜https://t.co/Vpac2q90CS
🧵1/9

[Amit Zohar @amit_zhr]

Thrilled to announce that our paper has been accepted for an Oral presentation at #ECCV2024!
See you in Milan!
With @urielsinger, @YKirstain, @ShellySheynin, @adam_polyak90, @deviparikh, and @taigman

[Kevin Li @kevinyli_]

Attention is all you need; at least the matrices are, if you want to distill Transformers into alternative architectures, like Mamba, with our new distillation method: MOHAWK!

We also release a fully subquadratic, performant 1.5B model distilled from Phi-1.5 with only 3B tokens!

[Alex Dimakis @AlexGDimakis]

Datacomp-LM (DCLM) was presented today in ICLM FOMO workshop. DCLM is a data-centric benchmark for LLMs. It is also the state of the art open-source LLM and the state of the art open training dataset.

Probably the most important finding is that data curation algorithms that work at 400M scale and predict performance at bigger sizes (at least up to 7B models). That means we can do data curation science by developing data curation algorithms at small scale and extrapolating.

You don't have to build the whole ship and throw it in the ocean to see to check if it floats.
You can build a minature ship and test in the bathtub.

[Achal Dave @achalddave]

Training DataComp-LM models meant we needed fast training code: here's a quick summary of how we sped up training in OpenLM by 60%, reducing costs by ~40%!

[Vaishaal Shankar @Vaishaal]

DCLM models keep on coming! This time we release (by far) the best open-data 1B model!

[Maor Ivgi @maorivg]

It’s incredible to witness the power of high-quality data. Today, we’re excited to share a fully open-source 1B model that outperforms many larger ones.

[Achal Dave @achalddave]

Excited to share our new-and-improved 1B models trained with DataComp-LM!

- 1.4B model trained on 4.3T tokens
- 5-shot MMLU 47.5 (base model) => 51.4 (w/ instruction tuning)
- Fully open models: public code, weights, dataset!

[Achal Dave @achalddave]

Excited to share our new-and-improved 1B models trained with DataComp-LM!

- 1.4B model trained on 4.3T tokens
- 5-shot MMLU 47.5 (base model) => 51.4 (w/ instruction tuning)
- Fully open models: public code, weights, dataset!

[Ori Yoran @OriYoran]

Can AI agents solve realistic, time-consuming web tasks such as “Which gyms near me have fitness classes on the weekend, before 7AM?"

We introduce AssistantBench, a benchmark with 214 such tasks.

Our new GPT-4 based agent gets just 25% accuracy!

https://t.co/S3ok3Zmi0T

[Akash Shetty @akashlives]

🚀 Exciting news! @Apple has released its own open-source LLM, DCLM-7B. Everything is open-source, including the model weights and datasets.

💡Why should you be excited?

1. The datasets and tools released as part of this research lay the groundwork for future advancements in optimizing training datasets for superior language model performance.

2. The data curation process is a must-study for anyone looking to train a model from scratch or fine-tune an existing one.

Research Page - https://t.co/X2Er4aAGfZ

#AI #MachineLearning #OpenSource #Apple #LLM #DataScience