Manan Dey

[Niklas Muennighoff @Muennighoff]

we just released: MVEB: Massive Video Embedding Benchmark 🎥

with more ai-generated videos, good video embeddings may become key, as you cant just grep through videos like for text..

[Shayne Longpre ✈️ ICML 🇰🇷 @ShayneRedford]

Thrilled our global data ecosystem audit was accepted to #ICLR2025!

Empirically, we find:

1⃣ Soaring synthetic text data: ~10M tokens (pre-2018) to 100B+ (2024).

2⃣ YouTube is now 70%+ of speech/video data but could block third-party collection.

3⃣ <0.2% of data from Africa/South America.

1/

[Caiming Xiong @CaimingXiong]

Testing LLMs' reasoning skills is tough—human evaluations are expensive, data contamination is common, and LLM judges can be biased. We propose StructTest, the first benchmark that checks how well LLMs follow complex instructions and create structured outputs. It uses a rule-based evaluator that’s easy to adapt to new tasks. StructTest is unbiased, cheap, hard to cheat and highly scalable.

By testing structured outputs in areas like Summarization, Code, HTML, and Math—and evaluating 17 top LLMs—StructTest proves to be a challenge even for models like Deepseek-V3/R1 and GPT-4o. It’s also highly correlated with ChatBot Arena (~93%) and MMLU (>96%), making it a solid way to measure reasoning skills.

Code & Data: https://t.co/5urKBaXJLT
Paper🔗: https://t.co/ASLGIuSR0F

[Shayne Longpre ✈️ ICML 🇰🇷 @ShayneRedford]

✨New Report✨ Our data ecosystem audit across text, speech, and video (✏️,📢,📽️) finds:

📈 Rising reliance on web, synthetic, and YouTube data.

🛑 80%+ datasets carry hidden restrictions.

🌍 Relative representation in languages and creators has not improved for 10+ yrs.

We're delighted to see this study covered by @Melissahei in the @techreview: https://t.co/6oJREDT3R9

1/

[Manan Dey @manandey]

@Muennighoff @Stanford @karpathy Congrats, Niklas!

[Shayne Longpre ✈️ ICML 🇰🇷 @ShayneRedford]

✨New Preprint ✨ How are shifting norms on the web impacting AI?

We find:

📉 A rapid decline in the consenting data commons (the web)

⚖️ Differing access to data by company, due to crawling restrictions (e.g.🔻26% OpenAI, 🔻13% Anthropic)

⛔️ Robots.txt preference protocols are ineffective

These precipitous changes will impact the availability and scaling laws for AI data, affecting coporate developers, but also non-profit and academic research.

🔗 https://t.co/NFSd9HYBlk

1/

[Manan Dey @manandey]

@KassnerNora @DeepMind Congratulations, Nora!

[BigCode @BigCodeProject]

Introducing: 💫StarCoder

StarCoder is a 15B LLM for code with 8k context and trained only on permissive data in 80+ programming languages. It can be prompted to reach 40% pass@1 on HumanEval and act as a Tech Assistant.

Try it here: https://t.co/4XJ0tn4K1m

Release thread🧵

[BigCode @BigCodeProject]

Announcing a holiday gift: 🎅SantaCoder - a 1.1B multilingual LM for code that outperforms much larger open-source models on both left-to-right generation and infilling!

Demo: https://t.co/jypW1SF75t
Paper: https://t.co/YV3pzUbYOr
Attribution: https://t.co/aE22O79Hnp

A🧵:

[Shanya Sharma @evolvedeve]

✨Our work "How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contexts" got accepted at the Findings on EMNLP 2022!✨

Joint work with @manandey and our awesome mentor @koustuvsinha 🎉

[Manan Dey @manandey]

@koustuvsinha @MetaAI Congratulations, Koustuv!

[BigScience Research Workshop @BigscienceW]

BLOOM is here. The largest open-access multilingual language model ever. Read more about it or get it at
https://t.co/mE013I62In
https://t.co/KrBRVklXLf

[Koustuv Sinha @koustuvsinha]

New paper alert! 🎉 Turns out you can reduce the gender biases your translation models just using relevant contexts, purely during inference! Checkout this cool work led by @evolvedeve and @manandey! https://t.co/UidZUV8avf [1/4]

[Saulnier Lucile @LucileSaulnier]

🧐🕵️I am looking for the best possible open source tool to do memory profiling!

I would like to know what part of my python code is causing these memory usage spikes that don't necessarily come from the Python interpreter.

Looking forward to reading your recommendations! 🤗

[BigScience Research Workshop @BigscienceW]

We are releasing PromptSource, a toolkit for creating, sharing, and using natural language prompts.

We used it to create the largest open-source collection of English prompts: 2,000 prompts for 170 datasets!

📄 https://t.co/FO43dQFJlg
💻 https://t.co/IpBvBp0JS3

[Sabrina J. Mielke @sjmielke]

Tokenization—the least interesting #NLProc topic? Hell no! We, members of the @BigScienceW tokenization group are proud to present:

✨Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP✨
https://t.co/dpcw5VP14D

What's in it? [1/10]

[Victor Sanh @SanhEstPasMoi]

We’ve seen crazy interest in T0++ (pronounced "T Zero Plus Plus"), and almost 10’000 queries to the model since we announced it 3 days ago.

Probably the most hilariously decisive prediction from the model (courtesy of @_philschmid):
1/N

[BigScience Research Workshop @BigscienceW]

First modeling paper out of BigScience is here!

T0 shows zero-shot task generalization on English natural language prompts, outperforming GPT-3 on many tasks, while being 16x smaller!

Model: https://t.co/QvEaqkfmgk
Repo: https://t.co/IpBvBp0JS3
Paper: https://t.co/wTQz0WOYl0

[Manan Dey @manandey]

@koustuvsinha Thanks a lot, Koustuv for being an amazing mentor and your guidance throughout! 😀

[Shanya Sharma @evolvedeve]

Hi #NeurIPS2020! I and @manandey will be presenting our poster on *Evaluating Gender Bias in NLI* at the Workshop on Dataset Curation and Security today (11th Dec) at 2:30 PM EST. Drop by if you're around :)
cc: @koustuvsinha

Gather Town (Poster 19)
https://t.co/ASs7AyKqOe