UMassNLP

Natural language processing group at UMass Amherst @umasscs. Led by @thompson_laure @MohitIyyer @brendan642 @andrewmccallum #nlproc

[Kalpesh Krishna @kalpeshk2011]

@GoogleAI Paper's HuggingFace page: https://t.co/R9UVdny52v

[Prateek Yadav @prateeky2806]

Given that I have been closely working with @tuvllms and @kalpeshk2011, I can say that he is extremely well read, hard working and this paper is amazing. People should definitely check out the FLAMe as this is going to be impactful.

[Tu Vu @tuvllms]

🚨 New @GoogleDeepMind paper 🚨

We trained Foundational Large Autorater Models (FLAMe) on extensive human evaluations, achieving the best RewardBench perf. among generative models trained solely on permissive data, surpassing both GPT-4 & 4o.

📰: https://t.co/FIPFiHwXyt
🧵:👇

[Kalpesh Krishna @kalpeshk2011]

Check out our new @GoogleAI paper: we curate a mixture of 5M human judgments to train general-purpose foundational autoraters.

Strong LLM-as-judge scores on RewardBench (87.8%), and highest perf among baselines on LLMAggreFact + 6 other benchmarks!

📰 https://t.co/oUN4hDeNWx
👇

[Mohit Iyyer @MohitIyyer]

So proud to have hooded my first five PhDs today: @tuvllms, @kalpeshk2011, @simeng_ssun, @mrdrozdov, and Nader Akoury. Now, they're either training LLMs at Google, Nvidia, and Databricks, or staying in academia at Virginia Tech and Cornell. Excited to watch their careers blossom!

[Ankita Gupta @Anki98765]

Check out ezCoref, our open-source tool for easy coreference annotation across languages/domains.
Demo: https://t.co/KIJuYh2aeS
Re-annotation study via ezCoref reveals interesting deviations from prior work.
📜https://t.co/v5tcRck0k9

#CRAC2023 @emnlpmeeting Dec 6, 2:50PM
🧵👇

[Tu Vu @tuvllms]

📢 🌟PhD Openings🌟:

I am recruiting PhD students this cycle at Virginia Tech. If you want to dive into:

- in-context learning & tool-use LLMs
- instruction tuning
- parameter-efficient transfer learning
- few-shot learning

please apply by Dec 15!

👉https://t.co/HiFjcIPSak

[Andrew Drozdov @mrdrozdov]

✨ New Paper ✨

Deep dive on demonstrations to enhance LLM-based passage ranking 🚀 insights for pointwise ranking using query likelihood 🚀

https://t.co/KLga6EEx19

[Tu Vu @tuvllms]

📢 Want to adapt your outdated LLM to our ever-changing world? 🌏

Check out our code for FreshPrompt at https://t.co/TZcJligS3Q.
Colab: https://t.co/ZeKbsjg8n8.

🙏 We are grateful to @serp_api for their generous sponsorship of 5000 searches for FreshPrompt's users.

[Jason Wei @_jasonwei]

Nice paper by Tu Vu on factuality in LLMs: https://t.co/ju4xEJ1D1i, enjoyed contributing in a minor role to it while I was at Google.

The main takeaway for me is that most factuality benchmarks for LLMs don't really take into account the fact that many types of knowledge changes, and usually isn't accounted for in benchmarks.

For instance, TriviaQA is a benchmark that was pretty popular in NLP. I evaluated GPT-4 on it and compared GPT-4's answers with gold labels, and got an agreement rate of something ~95%. And when I manually looked at the disagreements, often the gold labels were wrong because the information had changed.

One question asked "Who plays Noah Newman on The Young and the Restless?" The answer was Robert Adamson from 2012-2020, when the benchmark was created, but since then it has been played by Rory Gibson. So to make a good benchmark, you have to account for a lot of factors in factuality, including whether some knowledge is permanent or if it may change. So to get factuality really right, you will actually need a fair amount of hedging.

[Yapei Chang @ ICML 2026 @YapeiChang]

Can LLMs summarize books exceeding their context windows? We design an evaluation protocol for collecting fine-grained human judgments on LLM-generated summaries & propose BooookScore, a reference-free automatic metric for narrative coherence.

https://t.co/xNXToBzt9r

🧵below:

[Quoc Le @quocleix]

A weakness of LLMs is that they don’t know recent events well. This is nice work from Tu developing a benchmark (FreshQA) to measure factuality of recent events, and a simple method to improve search integration for better performance on the benchmark.

[Tu Vu @tuvllms]

🚨 New @GoogleAI paper:

🤖 LLMs are game-changers, but can they help us navigate a constantly changing world? 🤔

As of now, our work shows that LLMs, no matter their size, struggle when it comes to fast-changing knowledge & false premises.

📰: https://t.co/4AkLERtw3q
👇

[Mohit Iyyer @MohitIyyer]

Evaluating the factuality of LLMs is tricky: what if they answer a question correctly but also generate a bunch of unrelated made-up stuff? We eval LLM answers to our new FreshQA dataset in both a "strict" (no made up stuff) and "relaxed" setting, see the paper for more!

[Sheshera Mysore @MSheshera]

I’m at #sigir2023 and presenting our work on interactively controllable personalisation! Come listen in room 101 between 11-12.30!

[brendan o'connor @brendan642]

Reminder - for the terrific interdisciplinary Text as Data conference, abstract submissions coming up - due Aug 4!

https://t.co/BjegNt4qpv

It's a great, small, non-archival conference to discuss emerging work with folks across social sciences, humanities, and computer science.

[UMassNLP @UMass_NLP]

In Prize-winning Paper, UMass Amherst Computer Scientists Release Guidelines for Evaluating AI-Generated Text : UMass Amherst https://t.co/eVQrU8GdzO

[Mohit Iyyer @MohitIyyer]

Huge congrats @tuvuumass, who just became my first graduated PhD student!! He'll be starting his own group soon @VT_CS, so prospective PhD applicants interested in topics like multitask/multimodal transfer learning, or param-efficient LLM adaptation: def apply to work with him!

[Tu Vu @tuvllms]

Moving forward, I will be splitting my time as a research scientist at @GoogleAI and an assistant professor @VT_CS.

I will also be recruiting Ph.D. students starting in Fall 2024 to work on effective and efficient transfer learning in the era of LLMs, please come join me!

[Tu Vu @tuvllms]

I would also like to thank all of my labmates @UMass_NLP and friends at @UMassAmherst, my mentors and collaborators at @GoogleAI and @MSFTResearch, and my family and friends all over the world who gave me support and encouragement throughout my Ph.D. journey.