Mathieu Ravaut

ML, NLP, Quant research.
Graduate from @ntunlpsg | @AstarI2r | @uoftcompsci | @centralesupelec. Currently QR at ADIA.

[Mathieu Ravaut @MatRavox]

@HanSineng @YaleEngineering @Stanford Congrats!

[Caiming Xiong @CaimingXiong]

As large language models increasingly serve as judges for evaluating other models during both development and deployment, most existing benchmarks still focus on non-contextual tasks like chat completions or logical reasoning. Our team present ⚖️ ContextualJudgeBench ⚖️ — a benchmark designed to evaluate LLM-as-judge capabilities in context-rich scenarios such as RAG-based QA and summarization.

In real-world business applications, context is everything—AI systems must provide accurate and timely information. But assessing their performance is challenging: judge models need to interpret both the context and the question, then evaluate accuracy, justification for refusals, and overall quality, including completeness and conciseness.
📘 Paper: https://t.co/7uH3DQA6q7
📊 Data: https://t.co/h5xyNW1zOl
💻 Code: https://t.co/K1G2Y6zTlw

[Caiming Xiong @CaimingXiong]

Testing LLMs' reasoning skills is tough—human evaluations are expensive, data contamination is common, and LLM judges can be biased. We propose StructTest, the first benchmark that checks how well LLMs follow complex instructions and create structured outputs. It uses a rule-based evaluator that’s easy to adapt to new tasks. StructTest is unbiased, cheap, hard to cheat and highly scalable.

By testing structured outputs in areas like Summarization, Code, HTML, and Math—and evaluating 17 top LLMs—StructTest proves to be a challenge even for models like Deepseek-V3/R1 and GPT-4o. It’s also highly correlated with ChatBot Arena (~93%) and MMLU (>96%), making it a solid way to measure reasoning skills.

Code & Data: https://t.co/5urKBaXJLT
Paper🔗: https://t.co/ASLGIuSR0F

[Mathieu Ravaut @MatRavox]

@XueFz @m__dehghani Yes for people who worked 5-10 years in industry between their Masters and PhD

[Mathieu Ravaut @MatRavox]

@meetdavidwan Nice work @meetdavidwan

[Mathieu Ravaut @MatRavox]

@0x_ale Can you elaborate on getting SG passport in 2-6 years?

[Mathieu Ravaut @MatRavox]

@vijaypradwi @NTU_ccds Congrats!

[Mathieu Ravaut @MatRavox]

@agihippo It's Noam Shazeer so it could be

[Mathieu Ravaut @MatRavox]

@XueFz Yes, research-induced workaholism is real

[Mathieu Ravaut @MatRavox]

@agihippo No I am serious.
SG has IMDA, AI-SG, etc
The EU has GDPR and huge taxes 🥲
At least SG is pushing forwards not backwards!

[Mathieu Ravaut @MatRavox]

Checkout our analysis paper, to be featured in ACL main conference:
https://t.co/k8H9AMN9s3
We investigate the "middle-curse" exposed by the LITM paper for the specific task of abstractive summarizarion. In brief: the middle curse is very much an issue ! w @JotyShafiq @AixinSG

[Mathieu Ravaut @MatRavox]

@SinclairWang1 @CaimingXiong @SinclairWang1 thanks for sharing, we will include it in the next version of the paper

[Shafiq Joty @JotyShafiq]

Hope this paper and the accompanying library become handy to the LLM community.

[Caiming Xiong @CaimingXiong]

Are your LLMs highly accurate, or simply contaminated?
As the race to build the best LLM intensifies, clean evaluation is becoming more important than ever, yet contaminated LLMs and benchmarks obfuscate the real performance of models.
Checkout our new work (comprehensive survey + library) at NTU-NLP lab & Salesforce Research on the critical issue of contamination detection in LLMs, cc @ntunlp @MatRavox @D_Boss001 @HailinChen3 @XingxuanLi @RuochenZhao3 @FangkaiJiao @qcwntu @CaimingXiong @JotyShafiq
Paper:
https://t.co/16jXQ1hNXk
Library:
https://t.co/vNgXrto1XJ

[Criteo AI Lab @CriteoAILab]

We are excited to share that some of our Criteos got the "Best Paper Award" at #EACL2024 👏🎉
Congrats @florian_flb @songdng @MatRavox @AlexAllauzen , Nancy F. Chen,
@albertlumbreras @LaureSoulier, Vincent Guigue and Patrick Gallinari.🏆
@eaclmeeting
📜 https://t.co/x0tSs06w0p

[Mathieu Ravaut @MatRavox]

Checkout our paper on SSMs for long-input summarization! Happening today at EACL 👇
Congrats to @florian_flb @songdng

[Song @songdng]

🤔 Ever wondered how to summarize an entire book 𝑎𝑙𝑙 𝑎𝑡 𝑜𝑛𝑐𝑒? We delve into this question in our recent paper “𝐋𝐎𝐂𝐎��𝐓: 𝐒𝐭𝐚𝐭𝐞-𝐒𝐩𝐚𝐜𝐞 𝐌𝐨𝐝𝐞𝐥𝐬 𝐟𝐨𝐫 𝐋𝐨𝐧𝐠 𝐃𝐨𝐜𝐮𝐦𝐞𝐧𝐭 𝐀𝐛𝐬𝐭𝐫𝐚𝐜𝐭𝐢𝐯𝐞 𝐒𝐮𝐦𝐦𝐚𝐫𝐢𝐳𝐚𝐭𝐢𝐨𝐧” (#EACL2024). 🧵 (1/8)

[Mathieu Ravaut @MatRavox]

To be featured at EACL 2024

[Mathieu Ravaut @MatRavox]

Ever wondered how to summarize an entire book *all at once* with a deep learning model?
Check out our latest paper on summarization:
https://t.co/aRt0EuUPld
Through State Space models, we can scale the input to 600k tokens (!)