Hugo Malard

PhD student in multimodal learning for audio understanding at @telecomparis.
Ex-intern @AIatMeta

[Hugo Malard @Mldhug]

You want to give audio abilities to your VLM without compromising its vision performance? You want to align your audio encoder with a pretrained image encoder without suffering from the modality gap?
Check our #NeurIPS2024 paper with @michelolzam @Steph_lat and Slim Essid

[Victor Letzelter @ ICML 2026 @VLetzelter]

This is the final work from my PhD, in collaboration with @abursuc @Mldhug @MathieuFontai19 @.grichard @.slimessid @ptrkprz at @valeoai and @telecomparis

Our poster presentation is Thu, Jul 9, 2026 • 10:30 AM – 12:15 PM KST.

[Salah Zaiem @salah_zaiem]

We are looking for audio and speech generation people, in Zurich, Paris or London to join our team at Google Deepmind. We build cutting-edge speech, music and audio (also audio-visual) generation capabilities. Reach out to Jason or me if interested. Retweets very appreciated !

[Alfredo Canziani @alfcnz]

An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment

[Hugo Malard @Mldhug]

If you want to learn more about audio-visual alignment and how to use it to give audio abilities to your VLM, stop by our @NeurIPSConf poster #3602 (East exhibit hall A-C) tomorrow at 11am!

[Hugo Malard @Mldhug]

You want to give audio abilities to your VLM without compromising its vision performance? You want to align your audio encoder with a pretrained image encoder without suffering from the modality gap?
Check our #NeurIPS2024 paper with @michelolzam @Steph_lat and Slim Essid

[Michel Olvera @michelolzam]

Great talk today by @LiuHaohe at the @tp_adasp group on Latent Diffusion Models (LDMs) as versatile audio decoder! Walked us through diffusion basics, AudioLDM for text-to-audio, audio quality enhancement, and neural codecs!

[Hugo Malard @Mldhug]

@michelolzam @Steph_lat Take a look at it here: https://t.co/QBBJJwUngY

[Hugo Malard @Mldhug]

You want to give audio abilities to your VLM without compromising its vision performance? You want to align your audio encoder with a pretrained image encoder without suffering from the modality gap?
Check our #NeurIPS2024 paper with @michelolzam @Steph_lat and Slim Essid

[arXiv Sound @ArxivSound]

``An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment,'' Hugo Malard, Michel Olvera, St\'ephane Lathuiliere, Slim Essid, https://t.co/3n53Rr95fX

[Salah Zaiem @salah_zaiem]

Given a number of ASR models of different sizes, how can I allocate an audio sample to the smallest one that will be good enough ? @Mldhug worked on this question during his internship, and ended up with interesting conclusions you will find in our paper !

[Hugo Malard @Mldhug]

Check it out: https://t.co/hiNg3bsc3e

[Hugo Malard @Mldhug]

The preprint of our work (with @salah_zaiem and @AlgayresR) on sample dependent ASR model selection is available on arXiv!
In this paper we propose to train a decision module, that allows, given an audio sample, to use the smallest sufficient model leading to a good transcription

[Maksym Andriushchenko @maksym_andr]

It's really surprising how far one can go with *linear* predictors in the autoregressive setting.

Interesting theory and experiments on TinyStories: a linear model (with 162M params :-) ) can generate totally coherent text with few grammatical mistakes.

https://t.co/1BiZkgjxPm

[Puyuan Peng @PuyuanPeng]

Why is Whisper so robust to background noise? Not because Whisper suppresses them, but because Whisper 𝐮𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐬 them!

Check out the amazing work by Yuan Gong @YGongND. They reveal this emergent capability of Whisper, and get SOTA *simultaneous* ASR + audio tagging

[Jack Langerman (in sf) @jacklangerman]

Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language

Looks promising; I'll have to try and see if it stands upto some poking ;-)

Love that they get around the need for multimodal training.

https://t.co/dM2udO8Sze
https://t.co/N2L7PZRNnD

[AK @_akhaliq]

DreamDiffusion: Generating High-Quality Images from Brain EEG Signals

paper page: https://t.co/IN9nHz4q3m

paper introduces DreamDiffusion, a novel method for generating high-quality images directly from brain electroencephalogram (EEG) signals, without the need to translate thoughts into text. DreamDiffusion leverages pre-trained text-to-image models and employs temporal masked signal modeling to pre-train the EEG encoder for effective and robust EEG representations. Additionally, the method further leverages the CLIP image encoder to provide extra supervision to better align EEG, text, and image embeddings with limited EEG-image pairs. Overall, the proposed method overcomes the challenges of using EEG signals for image generation, such as noise, limited information, and individual differences, and achieves promising results. Quantitative and qualitative results demonstrate the effectiveness of the proposed method as a significant step towards portable and low-cost ``thoughts-to-image'', with potential applications in neuroscience and computer vision.

[Wei-Ning Hsu @mhnt1580]

Super excited to finally launch Voicebox🤩, the most versatile speech generative model ever💬👄

Demo page: https://t.co/PPhR9KoXIS

[Lucas Beyer (bl16) @giffmana]

Who killed non-contrastive image-text pretraining? @AlecRad and @_jongwook_kim with the below Fig2 in CLIP.

Who collected the 7 Dragonballs and asked Shenron to resurrect it? Yours truly, in this new paper of ours.

Generative captioning is not only competitive, it seems better!

[Orr Zohar @orr_zohar]

Thanks for tweeting, @AK!  We’re super excited about the future of text-only vision model selection! 🙏

@MarsScHuang
@kcjacksonwang
@syeung10

[AK @_akhaliq]

Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration

paper page: https://t.co/qtWaUKGpV8

Although instruction-tuned large language models (LLMs) have exhibited remarkable capabilities across various NLP tasks, their effectiveness on other data modalities beyond text has not been fully studied. In this work, we propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual, audio, and textual information. Macaw-LLM consists of three main components: a modality module for encoding multi-modal data, a cognitive module for harnessing pretrained LLMs, and an alignment module for harmonizing diverse representations. Our novel alignment module seamlessly bridges multi-modal features to textual features, simplifying the adaptation process from the modality modules to the cognitive module. In addition, we construct a large-scale multi-modal instruction dataset in terms of multi-turn dialogue, including 69K image instances and 50K video instances. We have made our data, code and model publicly available, which we hope can pave the way for future research in multi-modal LLMs and expand the capabilities of LLMs to handle diverse data modalities and address complex real-world scenarios.

[Sagar Vaze @Sagar_Vaze]

We'll present GeneCIS at #CVPR2023 (Highlight)

TL;DR: While most image representations are *fixed*, we present a general way to train and evaluate models which can adapt to different *conditions* on the fly.

Code: https://t.co/ufMQjw9Jcf
Project page: https://t.co/I5xYlVCS1H
🧵