Hao Dong

Incoming PI at ELLIS Institute Finland and TTAP at Tampere University | PhD @ ETH Zurich

[ELLIS @ELLISforEurope]

🏹 Job alert: Multiple PhD and Postdoc Positions in Multimodal Learning, Foundation Models and Agentic Systems at ELLIS Institute Finland and Tampere University

📍 Helsinki & Tampere 🇫🇮
⏰ 31 December
🔗 https://t.co/SU05VtEw5G

[Hao Dong @mjektd]

#PhD #Postdoc #Hiring #MachineLearning #AI #Multimodal #FoundationModels #ELLIS #TampereUniversity #Finland

[Hao Dong @mjektd]

We're hiring! 𝗣𝗵𝗗 & 𝗣𝗼𝘀𝘁𝗱𝗼𝗰 𝗽𝗼𝘀𝗶𝘁𝗶𝗼𝗻𝘀 in multimodal learning, foundation models and agentic systems.

I will be joining the ELLIS Institute Finland and Tampere University as a PI and TTAP! I'm now recruiting PhD students and postdocs to join my research group.

[Hao Dong @mjektd]

More details on the positions and how to apply: https://t.co/j3YsDLmVS2

Please feel free to share this post or tag anyone who might be interested!

[Hao Dong @mjektd]

Core research areas
• Multimodal Learning and Generative AI
• Reliable and Trustworthy Machine Learning
• Generalization, Adaptation, and Continual Learning of Foundation Models and Agentic Systems
• Applications in Robotics, Healthcare, Industrial Monitoring, and Beyond

[Hao Dong @mjektd]

Our Multimodal Intelligence Lab focuses on multimodal learning, generative AI, and vision-language models, with a specific emphasis on how foundation models and agentic systems can safely adapt and generalize to new environments.

[Hao Dong @mjektd]

Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study

https://t.co/9eE7J7oi6v
https://t.co/tDdU92yPEK

[Hao Dong @mjektd]

Happy to share that our survey paper on Multimodal Adaptation and Generalization has been accepted by 𝐓𝐏𝐀𝐌𝐈!

Advances in multimodal adaptation and generalization: From traditional approaches to foundation models

Paper: https://t.co/ht8tPHST4s
Code: https://t.co/G7oYjRy8cF

[Hao Dong @mjektd]

We review state-of-the-art methods, benchmarks, and applications across classification, segmentation, OOD detection, and more. Finally, we highlight open challenges and future research directions.

[Hao Dong @mjektd]

🚀 New Survey Paper Alert! 📷

Excited to share our latest work: "Adapting Vision-Language Models Without Labels: A Comprehensive Survey"

https://t.co/0t5rFhNpZd
https://t.co/RoCLfzPVD9

[Hao Dong @mjektd]

In this survey, we introduce the first taxonomy of unsupervised VLM adaptation based on the availability of unlabeled visual data:
1️⃣ Data-Free Transfer
2️⃣ Unsupervised Domain Transfer
3️⃣ Episodic Test-Time Adaptation
4️⃣ Online Test-Time Adaptation

[Hao Dong @mjektd]

Vision-Language Models, such as CLIP, have demonstrated impressive zero-shot capabilities; however, in real-world deployments, their performance can decline without adaptation. Gathering labeled data is costly, so unsupervised adaptation has emerged as a powerful alternative.

[Zhengzhong Tu @_vztu]

🚀 𝗧𝗵𝗿𝗶𝗹𝗹𝗲𝗱 𝘁𝗼 𝗮𝗻𝗻𝗼𝘂𝗻𝗰𝗲 𝗗𝗣𝗨 was 𝗮𝗰𝗰𝗲𝗽𝘁𝗲𝗱 𝗮𝘀 𝗮 𝗦𝗽𝗼𝘁𝗹𝗶𝗴𝗵𝘁 𝗮𝘁 𝗖𝗩𝗣𝗥 𝟮𝟬𝟮𝟱! 🎉✨

🌟 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗣𝗿𝗼𝘁𝗼𝘁𝘆𝗽𝗲 𝗨𝗽𝗱𝗮𝘁𝗶𝗻𝗴 (𝗗𝗣𝗨): A Breakthrough for Multimodal 𝗢𝘂𝘁-𝗼𝗳-𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻 𝗗𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻 (𝗢𝗢𝗗). Robust AI must identify unfamiliar inputs to avoid costly mistakes—but how do we tackle multimodal inputs like video, optical flow, and audio? 🤖🎬🎧

Our solution, DPU, dynamically updates class prototypes to capture intra-class variations, significantly boosting OOD detection performance. 🚦🔍

🎯 𝗞𝗲𝘆 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀:

• 💡 First to reveal and tackle 𝘪𝘯𝘵𝘳𝘢–𝘤𝘭𝘢𝘴𝘴 𝘷𝘢𝘳𝘪𝘢𝘣𝘪𝘭𝘪𝘵𝘺 challenges in multimodal OOD detection.
• ⚙️ 𝘗𝘭𝘶𝘨-𝘢𝘯𝘥-𝘱𝘭𝘢𝘺, 𝘮𝘰𝘥𝘦𝘭-𝘢𝘨𝘯𝘰𝘴𝘵𝘪𝘤 framework compatible with diverse OOD models.
• 📈 Achieves state-of-the-art results, improving Far-OOD detection performance by up to 80%!

🙌 Big thanks to my fantastic collaborators: Li Li, Huixian Gong, Hao Dong, Tiankai Yang, Yue Zhao, from University of Southern California and ETH Zürich.

🔗 Read the full paper: https://t.co/4BQxs4k3rl
🛠️ Code Available: https://t.co/OjlYyD5MVI

𝗟𝗼𝗼𝗸𝗶𝗻𝗴 𝗳𝗼𝗿𝘄𝗮𝗿𝗱 𝘁𝗼 𝗱𝗶𝘀𝗰𝘂𝘀𝘀𝗶𝗼𝗻𝘀 𝗮𝘁 𝗖𝗩𝗣𝗥 𝟮𝟬𝟮𝟱! 𝗟𝗲𝘁'𝘀 𝗽𝘂𝘀𝗵 𝘁𝗵𝗲 𝗯𝗼𝘂𝗻𝗱𝗮𝗿𝗶𝗲𝘀 𝗼𝗳 𝗺𝘂𝗹𝘁𝗶𝗺𝗼𝗱𝗮𝗹 𝗔𝗜 𝗳𝘂𝗿𝘁𝗵𝗲𝗿 𝘁𝗼𝗴𝗲𝘁𝗵𝗲𝗿! 💬✨

[Hao Dong @mjektd]

Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models

https://t.co/ht8tPHSleU
https://t.co/G7oYjRxAn7

[Hao Dong @mjektd]

These components enhance the model’s ability to distinguish unknown class samples during online adaptation by amplifying the entropy difference between known and unknown samples.

[Hao Dong @mjektd]

Happy to share our #ICLR2025 paper on Multimodal Open-set Test-time Adaptation!

https://t.co/1NkaM8MuwQ

[Hao Dong @mjektd]

To leverage this, we propose two key components: Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrepancy Optimization (AMP).

[Hao Dong @mjektd]

We introduce an Agree-to-Disagree (A2D) training algorithm, inspired by the Modality Prediction Discrepancy phenomenon. We also introduce a new outlier synthesis algorithm NP-Mix that explores broader feature spaces and complements A2D to strengthen OOD detection performance.