Emma Scharfman

ML for science 🤗 @huggingface
open science 🔍 + open data 📈 = breakthrough

DM me for any questions about science & HF 🤗
🔗

[Flavien CORONINI @FlavienC]

🚀 Hiring: Low-Level Senior SWE on the Xet storage team @huggingface.

📦 200PB+ of models and datasets behind the Hub. Rust, distributed systems, production ops.

You'd work on xet-core (open source) and the storage backend.

🌍 Remote, async, low process. Links to apply in MP👇

[Emma Scharfman @EmmaScharfmann]

Let your agent participate in Alzheimer’s research! Alzheimer’s affects more than 30 million people worldwide. The research community agrees that a gene variant (called APOE4) is a major risk factor, but still disagrees about why.

The evidence for the competing hypotheses already exists, but it’s scattered across the literature, making it hard to extract. MecCog is a collaborative challenge to build a structured map of that evidence.

We're running it with members of the BEACON consortium and the APOE4 research community. BEACON brings together pioneers in using data competitions to advance science, including leaders of CASP, the community challenge that helped catalyze breakthroughs such as AlphaFold, which was recognized with the 2024 Nobel Prize in Chemistry.

In this first phase, your agent finds precise, citable sources that provide evidence for leading hypotheses: papers, quotes, effect sizes, anything relevant. This is only phase 1 of a much bigger challenge, so stay tuned! 👀

🤗 Who can participate? Anyone! Just click “add your agent” and join the collaboration!
🗓️ Live now until 27 August 2026
🔗 https://t.co/qBKQvhrFT7

[Emma Scharfman @EmmaScharfmann]

Very cool research paper which focuses on making open source artifacts easier to run. It provides a complementary framework to FAIR to measure and hopefully reduce user-facing friction when using open source datasets: https://t.co/srfX5LxrqO

[Emma Scharfman @EmmaScharfmann]

@ajs6888 exactly, that's why we want to reduce this friction

[Emma Scharfman @EmmaScharfmann]

Open source models are great until you realize how hard they are to run yourself. I published a new blog post to make ECMWF's new AI model, AIFS, easy to use for anyone.

We built a compatibility patch so you can run inference on any GPU either using Hugging Face Jobs or your own hardware. In the blog post, we provide tutorials to run live weather forecasts, which can be used for research, for teaching, or just for fun. You can also use our demo space if you want to see it in action without running it yourself.

Thanks @ECMWF for making AIFS weights available on Hugging Face 🤗

[Emma Scharfman @EmmaScharfmann]

@regsky_ or maybe also hardware issue?

[Emma Scharfman @EmmaScharfmann]

@cgeorgiaw we can't all have the newest GPU's... it would make Nvidia too rich!

[Emma Scharfman @EmmaScharfmann]

Blog: https://t.co/2bqC0ocnQq
Code and tutorials: https://t.co/gJRDNVNEuD
AIFS model : https://t.co/Ke1lvBLkds

[Emma Scharfman @EmmaScharfmann]

"I’m not interested in having chatbots take over my writing and reviewing tasks. Writing helps me to realize what aspects I still don’t understand. Reviewing others’ work lets me put on my critical-thinking hat." Anna Hodshire, Assistant Professor at Colorado State University.
We debate about whether AI can review papers. But reviewing isn't just a service to the research community. It's also how we learn

[Hugging Face @huggingface]

AI security improves when organizations share research, tools and real-world experience.

We’re joining industry leaders, including @NVIDIA, in the Open Secure AI Alliance to help organizations identify and address software vulnerabilities and strengthen critical systems.

Learn more: https://t.co/AC6coLN3EV

[Emma Scharfman @EmmaScharfmann]

@NoeFlandre agreed! some try, but it's fairly rare in the pool of journals (or at least, the communication about it is obscure).

[Emma Scharfman @EmmaScharfmann]

How can journals ask researchers to review papers every week and expect them not to use AI?
Using AI in peer review is a real - and tricky! - debate
Instead of looking away while reviewers use AI anyway, why not build tools that actually help researchers? Very few journals seem interested in doing that... or am I missing something?

[Emma Scharfman @EmmaScharfmann]

@namankatyal14 I am sure (and hope) that journals are building tools, but I feel that the communication is terrible. As it's often the case, some journals are quite obscure in their strategy and in the meantime, I feel that researchers are lost.

[Emma Scharfman @EmmaScharfmann]

I am not even surprised... https://t.co/03sXSxULsG

[Anton Lozhkov @anton_lozhkov]

For over two years, the largest open code dataset was The Stack v2… until today.

🥞 The Stack v3 is out: the largest open code dataset ever released: 114 TB, 770 languages, 224M repositories, ~5T tokens of deduplicated and filtered source code. Fully open, no restrictively licensed code included.

The upgrade:
- v2 (2024): 68 TB raw -> 2 TB / ~550B tokens, 618 languages filtered
- v3 (2026): 114 TB raw -> 15.9 TB / ~5T tokens, 713 languages filtered

C++ x15, TypeScript x7.5, Rust x7, Python x4.8. Even the COBOL corner of GitHub got a bigger slice.

Part of that is two fresh years of open source. Part of it is a bug we found in v2's deduplication - story below. 👀

Things that make v3 different:
1. Contents inline. The #1 complaint about v2 was "cool dataset, where's the actual code?" v2 shipped file IDs from the Software Heritage graph, and fetching contents was a DIY treasure hunt. v3 is self-contained: sources embedded directly, one row = one repository. Download finishes -> you start training.
2. Fresh crawl. v2 was a 2023 snapshot of even older crawls. v3 is a direct re-crawl of GitHub at the latest commit, completed by August 2025: 224M repositories, 44B files. Forks were only included if they had 5+ stars.

Two ways in:
🥞 stack-v3-train - near-deduplicated, quality-filtered, PII-redacted, contents inline. Point load_dataset at it and go.
https://t.co/px2LW3v6VB
🏗 stack-v3-full - the entire 114 TB corpus as an HF Storage Bucket: every duplicate kept with cluster IDs, stubs for excluded files. Roll your own dedup, filters, and mixes.
https://t.co/SQzP5gOmzl

The record isn't really ours to claim - it's millions of everyone's repositories, neatly stacked. Thanks to every developer who keeps their code public, the BigCode community, and friends who built the infrastructure that made this possible!

[Emma Scharfman @EmmaScharfmann]

@DilumSanjaya totally, but that means changing the way researchers get recognition for their work (for tenure, PhD etc..)

[Emma Scharfman @EmmaScharfmann]

I read everywhere that with AI, researchers are writing and will be writing more papers. But what if AI makes writing so easy that quantity stops being important and quality finally becomes THE metric that counts and researchers can focus on what they actually like doing?

Some people seem to think so (and that's refreshing 🫧) → "AI can help scientists publish less" was published in Nature Astronomy a month ago. It's worth a read.

Instead of trying to keep up with an outdated system, can't we change the system?

[Emma Scharfman @EmmaScharfmann]

@cgeorgiaw totally agree with you! Instead of staying on the sideline, let's change our standards and build a better research system!

[Emma Scharfman @EmmaScharfmann]

@HarshilLodhiya That sounds quite discouraging indeed... but let's try to build a better system instead of letting it collapse!

[Emma Scharfman @EmmaScharfmann]

@HarshilLodhiya Indeed, there is a huge gap between AI tools for research and what journals think --> see https://t.co/khksMpDQk3

It don't think AI is good or bad for research, it's a tool, and we should build what help research (not what helps AI companies or journals, but research)