Jan Trienes

PhD candidate at University of Marburg.

[Jan Trienes @jantrienes]

I am excited to present our study on information salience in LLMs today at #ACL2025NLP (x4/x5, Tue, 16:00--17:30). Please come by if you are interested!

📝 Behavioral Analysis of Information Salience in Large Language Models
With @jschloetterer @jessyjli @SeifertChristin

[Jan Trienes @jantrienes]

Do you want to know what information LLMs prioritize in text synthesis tasks? Here's a short 🧵 about our new paper: an interpretable framework for salience analysis in LLMs.

First of all, information salience is a fuzzy concept. So how can we even measure it?

[Jessy Li @jessyjli]

🌟Job ad🌟 We (@gregd_nlp, @mattlease and I) are hiring a postdoc fellow within the CosmicAI Institute, to do galactic work with LLMs and generative AI! If you would like to push the frontiers of foundation models to help solve myths of the universe, please apply!

[Jan Trienes @jantrienes]

There is more results in the paper, so please have a look 🤗: https://t.co/ACawPxeOmY
This is joint work with @jschloetterer @jessyjli @SeifertChristin

[Jan Trienes @jantrienes]

Do you want to know what information LLMs prioritize in text synthesis tasks? Here's a short 🧵 about our new paper: an interpretable framework for salience analysis in LLMs.

First of all, information salience is a fuzzy concept. So how can we even measure it?

[Jan Trienes @jantrienes]

Finally, we consider if LLMs can introspect (= direct rate the salience of questions), if those direct ratings correlate with their behavior and with human perceptions of salience. Surprisingly, LLM behavior only weakly correlates in those settings.

[Greg Durrett @gregd_nlp]

🤔 Want to know if your LLMs are factual? You need LLM fact-checkers.
​
📣 Announcing the LLM-AggreFact leaderboard to rank LLM fact-checkers.
​
📣 Want the best model? Check out @bespokelabsai’s’ Bespoke-Minicheck-7B model, which is the current SOTA fact-checker and is cheap and fast to run.
​
LLM-AggreFact collects 11 datasets across NLP tasks covering grounded factuality. These datasets consist of 🤖 LLM responses ✏️ annotated with their hallucinations with respect to grounding documents. This includes question answering and summarization, including RAGTruth, TofuEval, ExpertQA, and more.
​
We benchmark 27 models on the task of detecting hallucinations.
​
Frontier LLMs are good at this task, but very expensive to use in real-world RAG pipelines! Bespoke's model is a step towards We invite progress on this benchmark to figure out what’s the smallest and fastest model we can get to achieve top scores!

[Jan Trienes @jantrienes]

Happy to share that our paper, InfoLossQA, has been accepted to #ACL2024 main 🥳 Thanks to all my great co-authors and looking forward to talking to you in Bangkok! #NLProc

[Jan Trienes @jantrienes]

When we (or LLMs) explain technical texts, those explanations can be vague or omit detail. How can we characterize such information loss and help lay readers intuitively recover it?

Excited to share InfoLossQA! 🧵

Paper: https://t.co/YxnKSCDrug
Website: https://t.co/9zyIE1Lec7

[Kyle Lo @kylelostat]

excited to share our contribution to open science of language models!

🐈‍⬛ all our data, weights, ckpts, code, etc
🐈 covers data curation, pretraining, adaptation, evaluation, etc

check out more deets in @soldni ‘s thread, technical reports out on arXiv shortly 😆

[Jan Trienes @jantrienes]

I was fortunate to visit @jessyjli at the University of Texas at Austin to work on this project. Thanks to @DAAD_Germany and @IkimUme for supporting this research visit.

[Jan Trienes @jantrienes]

We propose to use QAs to describe information loss. The Q asks for missing information, the A provides it. We release a dataset with 1,000 QA pairs highlighting information loss across 104 simplifications of clinical trials in medicine 🏥. View all annotations on our website.

[Jan Trienes @jantrienes]

Read the paper: https://t.co/YxnKSCDrug
Explore the dataset: https://t.co/9zyIE1Lec7

This is joint work with @sebajoed @jschloetterer @SeifertChristin @kylelostat @cocoweixu @byron_c_wallace @jessyjli

[Jan Trienes @jantrienes]

When we (or LLMs) explain technical texts, those explanations can be vague or omit detail. How can we characterize such information loss and help lay readers intuitively recover it?

Excited to share InfoLossQA! 🧵

Paper: https://t.co/YxnKSCDrug
Website: https://t.co/9zyIE1Lec7

[Jan Trienes @jantrienes]

@DercksenKoen @jschloetterer @SeifertChristin Hi @DercksenKoen, you can find it here: https://t.co/QDgRN9DOhp

[Jan Trienes @jantrienes]

Tomorrow I'll be presenting our work on creating a dataset for clinical text simplification at the TSAR-2022 workshop at #emnlp2022. Hope to see you there!

[M2L school @M2lSchool]

Final thanks to all #M2Lschool2022 participants, tutors, local organizers and volunteers at @unimib, sponsors @BancaSella @bendingspoons @Deepmind @Kosenlabs @ReplyULabs @unimib @WunThompson(Italy) Apple @Amplifon, and all our speakers! See you next time!