Chaoqun He

CS PhD @Tsinghua_Uni, @TsinghuaNLP

[Chaoqun He @Hothan01]

👉Key observations:

1️⃣Current models often struggle to balance timely responses with coherent global understanding.

2️⃣They may stay silent for much of the video, produce sparse captions, or fail to decide when a response should be triggered.

🧵5

[Chaoqun He @Hothan01]

🥳🙌Excited to introduce 🔥#Omni-DuplexEval🔥, a benchmark for evaluating real-time duplex omni-modal interaction.🚀

Models must not only understand streaming video/audio, but also decide when to respond.

Paper: https://t.co/k5Vq2NgCkf
Code/Data: https://t.co/YDgbMwFkPT

🧵1

[Chaoqun He @Hothan01]

Our experiments reveal a substantial gap.

The best-performing model achieves only 39.6% overall, compared with 81.8% for real-time human performance.

On Proactive Reminder, the best model reaches only 20.0%.

🧵4

[OpenBMB @OpenBMB]

🚨 MiniCPM-V 4.0 is here! 🚨
✨Key features:
🏗️4.1B parameters
🧠Matches GPT-4.1-mini-20250414 on image understanding tasks in OpenCompass evaluation
⚡Runs locally on iPhone 16 Pro Max at 17.9 tokens/sec
69.0 score on OpenCompass (vs 65.2 from MiniCPM-V 2.6 8B)
🚀Small model, good performance 🔥
⭐GitHub:https://t.co/gtRJoHOlfd
🤗HF:https://t.co/BG8pGuSsPk
#AI #LLM #MLLMs #MiniCPM #EdgeAI

[OpenBMB @OpenBMB]

🚀 MiniCPM4 is here! 5x faster on end devices 🔥
✨ What's new:
🏗️ Efficient Model Architecture
- InfLLM v2 -- Trainable Sparse Attention Mechanism
🧠 Efficient Learning Algorithms
- Model Wind Tunnel 2.0 -- Efficient Predictable Scaling
- BitCPM -- Ultimate Ternary Quantization
📚 High-Quality Training Data
- UltraClean -- High-quality Pre-training Data Filtering and Generation
- UltraChat v2 -- High-quality Supervised Fine-tuning Data Generation
⚡ Efficient Inference System:
- https://t.co/OQA29fxLVm -- Light Lightweight and Efficient CUDA Inference Framework
- ArkInfer -- Cross-Platform Deployment Framework

📖 Technical Report: https://t.co/LbnE6Es1DP
🤗 Models: https://t.co/8YuCqKbk1q
⭐ GitHub: https://t.co/zaffsLsx5m
#AI #LLM

[Jiao Sun @sunjiao123sun_]

Mitigating racial bias from LLMs is a lot easier than removing it from humans!

Can’t believe this happened at the best AI conference @NeurIPSConf

We have ethical reviews for authors, but missed it for invited speakers? 😡

[Chaoqun He @Hothan01]

#ACL2024 Another work:🔍OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems

arXiv: https://t.co/Rc1ucxCViv
Hugging Face: https://t.co/IDzEei4Pie

📍Poster Session 5
⏰August 13 16:00 - 17:30

See you!🥳

[Chaoqun He @Hothan01]

I am now in Bangkok🇹🇭 for #ACL2024 🥳

I will present our work:🔍UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs

arXiv: https://t.co/Mc9ssPFp0E
GitHub: https://t.co/uBdV6D77th

📍Poster Session 3
⏰August 12 16:00 - 17:30

See you🥳

[Chaoqun He @Hothan01]

@OriolVinyalsML @JeffDean try OlympiadBench?

[Chaoqun He @Hothan01]

@YouJiacheng Thanks, 👉the results for En_COMP, Zh_COMP, and Zh_CEE.

[Chaoqun He @Hothan01]

🧵6
The key results are as follows:
1️⃣ #GPT4V only achieves 17.23%. #GPT4 gets 29.50% on text-only tasks. #OlympiadBench is more challenging.
2️⃣A huge gap between closed- and open-source models.
3️⃣ The challenge lies more on question-withimages, Physics and none-English text.

[Chaoqun He @Hothan01]

🥳🙌Excited to release 🔥#OlympiadBench🔥, an Olympiad-level bilingual multimodal scientific benchmark. The best-performing model, #GPT4V, attains an average score of 17.23%. Such a challenging benchmark🚀

🤖GitHub: https://t.co/qmDbWhNDHT
📎Arxiv: https://t.co/a4eU5TAex3

🧵1

[Chaoqun He @Hothan01]

🧵9
Such as, given a simple solution, #GPT4V may choose a more complex method to solve the problem.

[Chaoqun He @Hothan01]

🧵8
Mistake Analysis of #GPT4V

We manually sample and check 97 maths (55 for English and 42 for Chinese) and 67 physics Olympiad-level open-ended problems that #GPT4V fails, and analyze the type of mistakes, the overall results are as follows