David Anugraha

[David Anugraha @DavidAnugraha]

🌍 Current VLMs are fundamentally English-centric.

Switching to native language prompts causes small but consistent drops across all resource levels. 📉

Switching context to the native language is much worse, up to a 32-point drop for low-resource languages, even for models explicitly trained on multilingual data. 💥

We hope that M4-RAG can serve as a foundation for future RAG systems that reason robustly across languages, modalities, and cultural contexts! 🌐

#CPVR2026 #Multimodal #Multilingual #Multicultural #Retrieval #RAG #VLM #NLP #CV #Scaling (5/5)

[David Anugraha @DavidAnugraha]

🎉 I am excited to share that our work "M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG" has been accepted to CVPR 2026 Main Track! 🎉

Thank you to all of my collaborators: @patrickamadeus_ , Anshul Singh, Annie Lee, and @gentaiscool for all their efforts in creating this work! Please check out our work, especially if you are working on multimodal multilingual RAG, and looking forward to connecting with others in Denver!

TL;DR: We release M4-RAG, a massive multilingual multimodal benchmark (42 languages, 56 dialects, 189 countries) for retrieval-augmented VQA. Our results show that while RAG helps smaller VLMs, it often fails to scale to larger models and can even degrade performance. We also observe substantial performance drops for non-English queries and retrieved context, and provide analysis and recommendations for building more effective multilingual multimodal RAG systems.

Check out these quick links or this thread out!
📄 Paper: https://t.co/zCLN1mTxTz
💻 Code: https://t.co/fZSgj1nvFJ
🤗 Dataset: https://t.co/iQoC8NuKr0

#CPVR2026 #Multimodal #Multilingual #Multicultural #Retrieval #RAG #VLM #NLP #CV #Scaling (1/5)

[David Anugraha @DavidAnugraha]

🔍 The bottleneck is also integration, not only retrieval.

We measure two things: how often RAG preserves correct answers (retention ✅) and how often it fixes wrong ones (correction rate 🔧), both with respect to retrieval quality.

Larger models resist updating wrong answers even when given the right evidence. The fundamental challenge isn't only about finding relevant knowledge, but also getting models to actually use it. 🧱

#CPVR2026 #Multimodal #Multilingual #Multicultural #Retrieval #RAG #VLM #NLP #CV #Scaling (4/5)

[David Anugraha @DavidAnugraha]

🎉 I am excited to share that our work "mR3: Multilingual Rubric-Agnostic Reward Reasoning Models" has been accepted to ICLR 2026!

Same as last year, I am truly grateful that the super intense month-long rebuttal process with many more sleepless nights running experiments was worthwhile 🥲

A huge thanks to all of my collaborators: @HLeiTR , @Zilu_Tang_Peter , @ Annie Lee, @DerryTWijaya1 , and @gentaiscool for all their efforts in creating this work! I am also very thankful for the annotators who contributed to our project! Please check out our work, and looking forward to connecting with others at ICLR 2026!

📄 Paper: https://t.co/knao1T8teU
💻 Code: https://t.co/qjUljqR44U
🤗 mR3 Models: https://t.co/AbMOifIZ2q
🤗 mR3 Datasets: https://t.co/tu5lmkXF90

#ICLR2026 #mR3 #RewardModel #Rubric #Multilingual #Reasoning #Evaluation #NLP #LLM #Metrics #Alignment #Preference

[David Anugraha @DavidAnugraha]

📈R3 Performance

We fine-tuned our R3 models on top of Qwen3 (4B, 8B, 14B) and investigated their performance across different multilingual settings using the same aligned datasets.

🔥 On pairwise tasks like m-reward-bench, reward-bench, IndoPref, and MM-Eval, mR3 models outperform all baselines, including gpt-oss-120b, despite being 9x smaller.

📝 mR3 models also shine on multilingual general knowledge (INCLUDE-base-44), multilingual math reasoning (MGSM), and multilingual safety dataset (RTP-LX).

🌍 Even though English remains the strongest reasoning language, mR3 models can reason better in their native languages than the base model can in English, and they outperform direct language forcing by a wide margin!

Now, are you ready to try mR3? 😎
💬 Feedback & questions are welcome!

#mR3 #RewardModel #NLP #LLM #Metrics #Alignment #Preference (4/4)

[David Anugraha @DavidAnugraha]

🚀 Following R3, we are thrilled to introduce mR3: Multilingual Rubric-Agnostic Reward Reasoning Models 🎉

TL;DR: mR3 is a multilingual version of R3, a rubric-agnostic reward reasoning model trained on 72 languages, achieving the broadest language coverage in reward modeling to date. mR3 is able to surpass much larger models, such as GPT-OSS-120B, while being up to 9x smaller⚡!

📄 Paper: https://t.co/3HqoHv1DWT
💻 Code: https://t.co/6P8S8tv7JG
🤗 mR3 Models: https://t.co/y5iOQ2uGSn
🤗 mR3 Datasets: https://t.co/CtrvEFuaMy

Huge thanks to all of my collaborators: @HLeiTR,
@Zilu_Tang_Peter, @ Annie Lee, @DerryTWijaya1, @llamagrp, and @gentaiscool for all of their efforts into creating this work!

Check this thread out! #mR3 #RewardModel #NLP #LLM #Metrics #Alignment #Preference (1/4)

[David Anugraha @DavidAnugraha]

📦mR3 Dataset

mR3 is trained on a diverse and carefully curated mix of point-wise, pair-wise, and binary task datasets spanning 72 languages, totaling 100K high-quality samples.

🧪 How It’s Built
→ mR3 training data is carefully curated through:
• Initial curation across 125 languages from 3M samples in public datasets such as HelpSteer3-Preference, MATH-500 Multilingual, HumanEval-XL, MMMLU, and PolyGuardMix.
• High-quality reasoning generation on different multilingual settings strategies using gpt-oss-120b to improve supervision quality and further analysis.
• Quality filtering by task difficulty & reasoning depth.

#mR3 #RewardModel #NLP #LLM #Metrics #Alignment #Preference (3/4)

[David Anugraha @DavidAnugraha]

📈R3 Performance

We fine-tuned our R3 models on top of Qwen3 (4B, 8B, 14B) and Phi-4-Reasoning-Plus, using just 4K or 14K carefully filtered samples. We also tried a limited compute scenario by applying LoRA for efficient adaptation.

🔥 On pairwise tasks like RM-Bench and RewardBench, R3 models outperform all baselines, including the latest reasoning RM, RM-R1.

📝 R3 models also shine on point-wise scoring in summarization (XSUM), point-wise scoring in generic rubric assessment (FeedbackBench), and reasoning binary tasks (BBH Binary and MMLU-STEM Binary).

💪Despite minimal data and compute, R3 delivers strong generalization and rubric-flexible scoring across task types.

Now, are you ready to try R3? 😎
💬 Feedback & questions are welcome!

#R3 #RewardModel #Evaluation #NLP #LLM #Metrics #Alignment #Preference
(5/5)

[David Anugraha @DavidAnugraha]

⚠️New Paper Alert⚠️
Introducing R3: Robust Rubric-Agnostic Reward Models 🎉

TL;DR: R3 is a novel reward modeling framework that surpasses recent reasoning reward model performance on multiple benchmarks with 10× less data, all while providing interpretable, controllable, and generalizable scoring.

📄Paper: https://t.co/yB5YZVG4Em
👨‍💻Code: https://t.co/GZ1lQsjPvp
🤗R3 Models: https://t.co/77XWeWiVTE
🤗R3 Datasets: https://t.co/5MuUwY3cTl
🌐Website: https://t.co/DaUU4vvHd8

Huge thanks to all of my collaborators: @gentaiscool @Zilu_Tang_Peter @ljvmiranda @OptionsGod_lgd @rifqifrhnsyah @gkuwanto @DerryTWijaya1 @llamagrp who spent countless hours, late nights, and deep discussions that went into creating this work! 🙇‍♂️

Additionally, I am also extending my gratitude to @HLeiTR for his support! 🙏

Check this thread out! 🧵

#R3 #RewardModel #Evaluation #NLP #LLM #Metrics #Alignment #Preference
(1/5)

[David Anugraha @DavidAnugraha]

📦R3 Dataset

R3 is trained on a diverse, carefully curated mix of point-wise, pair-wise, and binary tasks dataset as shown in the diagram below

🗣️ General Chat & Instruction-Following
→ Datasets like Tulu, UltraFeedback, and Skywork Reward Preference provide open-domain instructions and preferences, including both point-wise and pairwise signals.

🧠 Reasoning Tasks
→ Math and code datasets such as Math-Step-DPO-10K and AceCodePair-300K focus on correctness and reasoning quality.

✅ Classification & Factuality
→ Includes NLI, summarization, and factuality data from GLUE, SuperGLUE, SummEval, FeedbackCollection, PreferenceCollection, and EVOUNA.

🧪 How It’s Built
→ R3 training data is carefully curated through:
• Diversity sampling across domains.
• High-quality reasoning generation using DeepSeek-R1 to improve supervision quality.
• Quality filtering by task difficulty & reasoning depth.

#R3 #RewardModel #Evaluation #NLP #LLM #Metrics #Alignment #Preference
(4/5)

[Genta Winata @gentaiscool]

⭐️We're thrilled to share that our paper WorldCuisines has been selected for the Best Theme Paper Award at NAACL 2025 @naaclmeeting! 🎉 A huge thank you to the reviewers and area chair for this incredible recognition — we’re truly honored.

Massive gratitude to all our amazing co-authors for the countless hours, late nights, and deep discussions that went into creating this high-quality dataset.

https://t.co/cxG0dDHDf2

We can't wait to present next week at NAACL! Catch us at our poster session (Wednesday, April 30) and Best Paper Award Session (Friday, May 2) for our oral presentation.

Check out the paper and project here:
🌐 https://t.co/wIXukNjBKk

Contributors:
@fredyhudi, @patrickamadeus_, @davidanugraha, @rifkiaputri, @zzeet, @ubaidalih, @auliaadilaa, @adamnohejl, @JunhoMyung00211, @aliceoh, @AnarSnowball, @faridlazuarda, @jcblaisecruz, @nedjmaou, @jodieyzhou, @AboladeDaud, @prajdabre1, @holylovenia, @SCahyawijaya, @bryanwilie92, @mrpeerat, @farizikhwantri, @gkuwanto, @llamagrp, @mv_zhukova, @EmmanueleChers1, @AlhamFikri, @davlanade, @tarowatanabe, @OptionsGod_lgd,@AyuP_AI, and many others who are not on X.

Acknowledgments:
@nayeon7lee, @Wenliang_Dai, @pascalefung
who helped and provided us insightful suggestions.

#nlproc #naacl2025 #worldcuisines

[Genta Winata @gentaiscool]

🌎 World Cuisines 🍟🌭🍕🍝🍙🍚🍢🍡

🎉 Big news! Thrilled to share that our WorldCuisines paper has been accepted by @naaclmeeting NAACL 2025!

⭐️ This is the first-ever large-scale high-quality benchmark for multilingual VQA on Food, covering 30 languages and dialects with over 1 million data points.

Check our paper, dataset, website, and leaderboard!
🤗 https://t.co/c0KlVHAdLr 📰 https://t.co/m0QUoauhpX
🌐 https://t.co/wIXukNjBKk
🏅 https://t.co/Js0l9SqGLH

Thank you for all collaborators @fredyhudi, @patrickamadeus_, @davidanugraha, @rifkiaputri, @zzeet, @ubaidalih, @auliaadilaa, @adamnohejl, @JunhoMyung00211, @aliceoh, @AnarSnowball, @faridlazuarda, @jcblaisecruz, @nedjmaou, @jodieyzhou, @AboladeDaud, @prajdabre1, @holylovenia, @SCahyawijaya, @bryanwilie92, @mrpeerat, @farizikhwantri, @gkuwanto, @llamagrp, @mv_zhukova, @EmmanueleChers1 @AlhamFikri @davlanade @tarowatanabe, and to the many others not on X, whose tireless contributions and language expertise were crucial to the success of this project.

#worldcuisines #nlproc #NAACL2025
#food #vlm #llava #nlvm #Pangea

[Genta Winata @gentaiscool]

📢 I am thrilled to announce that MetaMetrics has been accepted by @iclr_conf ICLR 2025! We pushed our limits to tackle the month-long rebuttal period, and it was well worth the effort. A heartfelt thank you to all the collaborators who believed in this unique idea and worked alongside me to make it a reality.

@DavidAnugraha @gkuwanto @llamagrp @DerryTWijaya1 and Lucky

Special thanks to @MingyangKevinZh for proofreading the paper and providing valuable suggestions.

Check our preprint: https://t.co/wOPdaCv7E3
Openreview: https://t.co/qXCWOGPBEB

#NLProc #metrics #metametrics #ICLR2025