Francesco Barbieri

health @MicrosoftAI

[Microsoft Copilot @Copilot]

At some point, we've all stared at a test result we didn't understand. Worn a device that tracked everything but revealed little. Most of us don't need more data — we need clarity, and that's what Copilot Health is for. Meet the team of clinical experts building the experience. https://t.co/msZ6IFGmAn

[Bill Shen @williamfshen]

📣 Our recent work at Meta: Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks

Rubric-based judging and reward modeling are powerful for open-ended tasks, but scaling requires controlled generation, not just free-form lists from an LLM.

[Shashwat Goel @ ICML 🇰🇷 @ShashwatGoel7]

In case you missed it over the holidays, we @AIatMeta released: Training AI Co-Scientists using Rubric Rewards.

- RL training, we open-source the training and eval datasets!
- Human study, finding 70% win-rate
- GPT-5's lead in science? Our eval quantifies the anecdotes!

[Shashwat Goel @ ICML 🇰🇷 @ShashwatGoel7]

🚨New paper: Training AI Co-Scientists using Rubric Rewards

In my recent internship at Meta Superintelligence Labs, I pursued an opinionated research bet: a general, scalable training recipe to improve AI at helping scientists achieve their research goals.

Motivation
Existing work on training AI for Science optimizes pre-defined, narrow scientific objectives with execution feedback in specially constructed environments (e.g. RLVR).

However, it's infeasible to learn from trial and error in many sciences. For e.g. medical research is hard to simulate digitally, and it is unethical to run clinical trials with suboptimal approaches proposed in early training.😬

Moreover, when pursuing a novel research goal, the primary intellectual challenge often lies in defining the experiment setup and objective itself. In the past year, I have increasingly used AI assistance for this (especially GPT-5) in my own research.

Of course, models often fail to follow some explicitly stated requirements, and sometimes propose bad design choices, but that is fine! The generated plans are still useful for brainstorming, and I can implement them with further refinement.

Method
This made us wonder🤔: how can we train models to be better at this task of generating research plans, given an open-ended research goal? For training, we need to collect a large number of research goals, and obtain fast verification signals. Human experts are expensive to access, and that wouldn't scale.

💡Equipped with the vast corpus of openly licensed scientific literature, and the recent success of RL, Synthetic Data Curation, and Rubrics, we propose a scalable recipe: Extract research goals and goal-specific grading rubrics from existing papers with an LLM, and use them for RL training.

Specifically, a frozen copy of the initial model rewards the plans generated during training using the goal-specific rubrics, checking seven general guidelines for parts of the plan relevant to each rubric item.

🤔Won't this lead to reward hacking? It will. At some point. But until then, improvements on the training reward might generalize to better research plans for humans. We are hoping the goal-specific rubrics, provided as privileged information to the grader, create a generator-verifier gap that improves research plan generation without external supervision.

The only way to find out? Perform a human study. We ask Machine Learning experts to compare plans generated by the finetuned vs initial Qwen3-30B model for ML research goals. This is slow and expensive, it required 45 minutes per annotation to carefully analyze plans, so we could only do this once at the end of the project for evaluation.

Results
Individual annotations are still noisy, as evaluating research plans is inherently subjective. But sure enough, there is non-trivial signal. The experts preferred (p < 0.01) our finetuned models plans for 70% research goals extracted from NeurIPS'24 / ICLR'25 Oral papers (top 1%) ✅

But only ML, and finetuned vs initial, is boring. Remember, the goal is generality. So we also finetuned Qwen3-30B on goals extracted from medical research, and new arXiv prerints spanning 8 domains. We use rubric evaluations with a jury of frontier models, which also allows us to compare many frontier models across domains. Notable findings:

1) In-domain finetuning leads to 12-22% relative improvements in scores across the three domains: arXiv, medical, and ML 📈
2) Significant cross-domain generalization, especially with the medical finetune improving on ML and new arXiv research goals. This might be evidence for our "generality" thesis 📊
3) Our 30B finetune matches much larger models like Grok-4-Thinking, but GPT-5-Thinking is a cut above the rest (consistent with my qualitative experience) 🤖

Limitations
Now of course, LLM-based evaluations, even with a jury and rubrics, are imperfect. But while the individual sample scoring is noisy, we hope for directionally correct results in aggregate, as the jury has positive alignment with human majority vote in our human study on ML. We think the grading scheme holds promise, as optimizing against a much weaker grader (30B), led to improvements in human preference.

This work has many such limitations, so treat it more like an early proof-of-concept. We candidly acknowledge them in our paper, and encourage you to scrutinize the details:

📜 https://t.co/eL9bjneqq9

Released Artefacts
The paper has many ablations and analyses:
- our appendix also has sample outputs across domains for vibe-checks, making it 119 pages!
- criteria-wise breakdown of performance evolution during training, thanks to our structured grading
- SFT on long-form plans worsened model performance
- training also improves Gemma, Llama models

🤗We release our train and test data on @huggingface. At a sample-level the data is noisy, and generated by Llama-4-Maverick. Still human experts approved 84% of the rubric items in ML so there's promise, and the same methodology will lead to better quality data as language models improve.

Overall, we think the potential of our approach is high: the scientific method is quite general, deep learning benefits from generality (transfer learning), and language models are amazing (better every month!). We hope approaches like this make LMs better at assisting researchers across diverse problem settings and scientific disciplines.

Some cool figures from the paper, and acknowledgements in thread🧵. I'm all ears to feedback on how we could've done things better!

1/3

[Dong-Ho Lee @Dongho_Lee_]

🤖 Even wonder why chatbots feel off?

We unveil REALTALK: 21 days of REAL human chats showing what AI misses: emotions, shifting personas, memory gaps.

https://t.co/cGrrMN71CE
https://t.co/SuW9sD7v4u

@Snap @saharaai @USC_ISI @nlp_usc
#LLM #EmotionalIntelligence

[Dimos @dimos_antypas]

1/ Social media connects people globally, but interpreting posts in different languages remains a significant challenge. We're tackling this with the X-Topic dataset! Our paper at EMNLP main! https://t.co/cPUxobUt3k🧵

[Zhihan Zhang @zhihz0535]

Excited to share that PLUG has been accepted to #ACL2024 main conference! Looking forward to the conference at 🇹🇭!!! Updated version at https://t.co/WSaOuIRQuj👀

[Agostina Calabrese 🦋 @agostina_cal]

If your #hatespeech research is motivated by a desire to make the web a safer place, you don't want to miss our new #ACL2024NLP paper:

✨"Explainability and Hate Speech: Structured Explanations Make Social Media Moderators Faster"✨

@Edin_CDT_NLP @EdinburghNLP @Snap #NLProc
1/6

[Hannah Rose Kirk @hannahrosekirk]

Today we're launching PRISM, a new resource to diversify the voices contributing to alignment. We asked 1500 people around the world for their stated preferences over LLM behaviours, then we observed their contextual preferences in 8000 convos with 21 LLMs https://t.co/P7nMKuvT23

[Adyasha Maharana @adyasha10]

Can LLMs keep track of very long conversations?

We evaluate 'conversational memory' of LLMs via 3 tasks on our dataset of multi-session multimodal dialogs --&gt; LLMs struggle to remember, reason over history, draw long-range temporal/causal connections

https://t.co/JrGP7imeMh

🧵

[Jose Camacho Collados @CamachoCollados]

“SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research” (EMNLP Findings, long, Sunday 9am online) by @dimos_antypas, @asahiushio, @fvancesco, @lnevespapers, @kia_8345, @espinosaanke and @jiaxin_pei

https://t.co/e5WQqtfeUQ

[Zhihan Zhang @zhihz0535]

🤨LLMs struggle to follow instructions in low-resource languages?
⚡️Introducing PLUG: leveraging pivot language in cross-lingual instruction tuning
📈Improved LLaMA-2 by 32% on 4 diverse languages!

Check out our new preprint at➡️https://t.co/WSaOuIRiEL

[Dimos @dimos_antypas]

🧵 Introducing "SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research"

[TweetNLP @tweet_nlp]

We are excited to officially announce 🚀Tweet Insights🚀, an online visualization platform to get temporal information from Twitter! 📈

Check it out at https://t.co/3JYCM8n2dE

[Lingbo Tong @lingbooo]

🎉 Excited to announce our 2nd KnowledgeNLP Workshop at #KDD2023! 🔍 Join us for an enlightening journey into the world of Knowledge Augmented #NLP. 🌐 Engage with leading experts, explore cutting-edge research, and unlock the power of big data. Don't miss out! #AI #DataScience

[Dan Lou @danielbloureiro]

Today is the day. I'm finally ready to share my next project: https://t.co/A1vHU2hhJt. I've been working on this ever since I took a break from research back in March. Let me now explain why I'm excited about this.

I believe the rise of privacy regulations (e.g., GDPR) along with the incredible recent progress of AI, has created an opportunity to try new approaches for marketing. Among those, developing AI Buyer Personas that sellers and brands can interact with seems most exciting to me.

Instead of collecting vast amounts of personal data in order to learn about consumer preferences and behaviours, which can be unethical and out of reach for small companies, we can now use LLMs to absorve, model, and anonymously distill that collective knowledge. I was first convinced this was possible by this excellent paper from Argyle et al 2023 (https://t.co/4aHmfuaBbD, before GPT-4 even), which introduced the concept of “silicon sampling” as a potential alternative to traditional survey-based human sampling of public opinion.

With https://t.co/5njjCp1bOS, I’ve developed a novel approach to simulate AI Buyer Personas from a wide range of models, data, and APIs. The first practical result from this effort is the ability to instantly match products to relevant personas, and generate custom ads for them. More applications will come if there’s interest.

Thoughts and feedback are deeply appreciated! Thank you for your support 😃

[asahi1992 @asahiushio]

Twitter question answering &amp; question answer generation are supported by tweetnlp now! We are actively expanding the scope of tweetnlp, so let us know any missing NLP tasks/datasets on Twitter!
#NLProc @Cardiff_NLP
Repo: https://t.co/jgU47HgqUG
HP: https://t.co/Q2Ky0rUjFF

[Jose Camacho Collados @CamachoCollados]

Tomorrow at #emnlp2022 we are presenting our TweetNLP platform (2pm, Demo session).

We have prepared some important improvements with respect to the previous version, check below! 👇

[Ever Evolving NLP @Evo_NLP]

EvoNLP is starting this Wednesday! #evonlp2022 #emnlp2022

Check out the schedule and invited talks at our website: https://t.co/7XwYIIDVgL

[Jose Camacho Collados @CamachoCollados]

🔥 SemEval 2023 task on Visual Word Sense Disambiguation 🔥

Given a (potentially ambiguous) word and some limited textual context, select the image with the intended meaning. #NLProc #ComputerVision

Training data available! https://t.co/hobrolsdIm