Raushan Turganbay

ML Engineer at HuggingFace

[alphaXiv @askalphaxiv]

Introducing autoresearch for arXiv papers

Change 'arxiv' to 'autoarxiv' in any paper URL

An agent deploys to resolve setup issues on the codebase, run a minimal reproduction, and estimate full replication cost. Read more below

[DailyPapers @HuggingPapers]

Microsoft research reveals information loss in Vision-Language Models

VLMs lose 40-60% semantic context after the projection step, distorting visual representations & impacting downstream tasks. See k-NNs diverge from fruit to mushrooms!

[Aran Komatsuzaki @arankomatsuzaki]

Apple presents AToken: A unified visual tokenizer

• First tokenizer unifying images, videos & 3D
• Shared 4D latent space (preserves both reconstruction & semantics)
• Strong across gen & understanding tasks (ImageNet 82.2%, MSRVTT 32.6%, 3D acc 90.9%)

[Arthur Zucker @art_zucker]

With the latest release, I want to make sure I get this message to the community: we are listening!

@huggingface we are very ambitious and we want `transformers` to accelerate the ecosystem and enable all hardwares / platforms!
Let's build AGI together 🫣
Unbloat and Enable!

[vLLM @vllm_project]

The @huggingface Transformers ↔️ @vllm_project integration just leveled up: Vision-Language Models are now supported out of the box!

If the model is integrated into Transformers, you can now run it directly with vLLM.

https://t.co/2CHOuZ1iqc

Great work @RTurganbay 👏

[Amir Zamir @zamir_ar]

We benchmarked leading multimodal foundation models (GPT-4o, Claude 3.5 Sonnet, Gemini, Llama, etc.) on standard computer vision tasks—from segmentation to surface normal estimation—using standard datasets like COCO and ImageNet.

These models have made remarkable progress; however, it is unclear exactly where they stand in terms of understanding vision in detail. Especially when it comes to tasks beyond question-answering. How well do they understand an object's segments or geometry?

Our analyses yield an assessment that is quantitatively and qualitatively detailed and is compatible with evaluations developed in the field of computer vision over the past decades.

Observed trends:

🔹 The foundation models consistently underperform task-specific SOTA models across all tasks. However, they are respectable generalists, which is remarkable as they are presumably trained primarily on image-text-based tasks.
🔹 They perform semantic tasks notably better than geometric ones.
🔹 GPT-4o performs the best among non-reasoning models, getting the top position in 4 out of 6 tasks.
🔹 Reasoning models, e.g., o3, show improvements in geometric tasks.
🔹 The 'image generation' models, e.g., GPT-40 Image Generation, which have been natively trained multimodally, exhibit quirks. E.g., hallucinated objects, misalignment between the input and output, etc.
🔹 While the prompting techniques affect performance, better models exhibit less sensitivity to variations in prompts. We control for the variance introduced by the prompting methods in our experiments.

🌐 Detailed analyses, visualizations: https://t.co/l8OVGMaX5V
⌨️ code: https://t.co/XufFNPWndi

🧵 1/n

[Arthur Zucker @art_zucker]

Holy... `transformers` reached 1B downloads 😭 thanks everyone for making this possible, what an amazing community

[Federico Baldassarre @BaldassarreFe]

DINOv2 meets text at #CVPR 2025! Why choose between high-quality DINO features and CLIP-style vision-language alignment? Pick both with dino.txt 🦖📖

We align frozen DINOv2 features with text captions, obtaining both image-level and patch-level alignment at a minimal cost. [1/N]

[Matthew Carrigan @carrigmat]

As of today, Transformers is deprecating TensorFlow and JAX and switching to being PyTorch-only!

The main reason is that other libraries were already doing a better job; we're still stuck on Keras 2, but KerasNLP supports loading our checkpoints with Keras 3 on any framework 🧵

[Lysandre @LysandreJik]

I have bittersweet news to share.

Yesterday we merged a PR deprecating TensorFlow and Flax support in transformers.

Going forward, we're focusing all our efforts on PyTorch to remove a lot of the bloating in the transformers library. Expect a simpler toolkit, across the board.

[SkalskiP @skalskip92]

I can finally map @NBA player's position from the camera perspective onto the court map

it's still a bit shaky... I'll smooth it out later

it's time to detect shooting motions and mark the shot location!

some of the code has already been migrated to: https://t.co/VK0RQFWud1

[merve @mervenoyann]

VLMS 2025 UPDATE 🔥

We just shipped a blog on everything latest on vision language models, including
🤖 GUI agents, agentic VLMs, omni models
📑 multimodal RAG
⏯️ video LMs
🤏🏻 smol models
..and more!

find it on the next one ⤵️

[𝚐𝔪𝟾𝚡𝚡𝟾 @gm8xx8]

LLM + vision encoder is the default multimodal setup, but is that optimal?

A scaling study across 457 models finds NO inherent advantage to late-fusion.
Early-fusion (no image encoder):
- Outperforms at smaller scales
- Trains faster, deploys easier
- Gains more w/ MoEs via modality-specific routing

Native multimodal > stitched pipelines

[𝚐𝔪𝟾𝚡𝚡𝟾 @gm8xx8]

Tokenize Image as a Set

TokenSet is a set-based image generation approach that replaces fixed-position latent codes with unordered tokens, allowing adaptive allocation based on regional complexity. It introduces a dual transformation to map sets to fixed-length integer sequences and proposes Fixed-Sum Discrete Diffusion to model discrete sets with summation constraints. The method supports semantic-aware representation and maintains generation quality without relying on sequential tokenization.

[Vaibhav (VB) Srivastav @reach_vb]

HOLY SHITT, Microsoft dropped an open-source Multimodal (supports Audio, Vision and Text) Phi 4 - MIT licensed! 🔥

> Beats Gemini 2.0 Flash, GPT4o, Whisper, SeamlessM4T v2
> Models on Hugging Face hub, integrated with/ Transformers!

Phi-4-Multimodal:

> Modalities: Integrates text, vision, and speech/audio
> Architecture: Uses "Mixture of LoRAs" to add modality-specific adapters without fine-tuning the base model
> Vision Modality: SigLIP-400M image encoder, 2-layer MLP projector, dynamic multi-crop strategy
> Speech/Audio Modality: 3-layer convolution, 24 conformer blocks, 80ms token rate
> Performance: Ranks first on OpenASR leaderboard, supports vision+language, vision+speech, and speech/audio tasks, outperforming larger models

Phi-4-Mini:

> Parameters: 3.8 billion
> Architecture: 32 Transformer layers, 3,072 hidden state size, Group Query Attention (GQA) with 24 query heads and 8 key/value heads
> Vocabulary: 200K tokens for multilingual support.
Training Data: High-quality web and synthetic data, emphasizing math and coding
> Performance: Outperforms similar-sized models and matches larger models (e.g., DeepSeek-Rl-Distill-Qwen-7B) on math and coding tasks

Training Pipeline:

> Language Training: Pre-training on 5 trillion tokens, post-training with function calling, summarization, and instruction-following data
> Multimodal Training: Vision training (4 stages), speech/audio training (2 stages), and joint vision-speech training
> Reasoning Training: Pre-trained on 60B CoT tokens, fine-tuned on 200K high-quality CoT samples, and DPO-trained on 300K preference samples

Vision Benchmarks:
> Outperforms Phi-3.5-Vision, Qwen2.5-VL, InternVL2.5, and matches Gemini and GPT-4o on tasks like chart understanding and OCR
> Vision-Speech Benchmarks: Significantly outperforms InternOmni and Gemini-2.0-Flash

Speech Benchmarks:

> ASR: Achieves SOTA on CommonVoice, FLEURS, and Open ASR Leaderboard, surpassing WhisperV3 and SeamlessM4T
> AST: Best performance on CoVoST2, comparable to GPT-4o on FLEURS
> Speech Summarization: First open-source model with this capability, close to GPT-4o in quality

Language Benchmarks:

> Outperforms similar-sized models (Llama-3.2, Ministral) and matches larger models (Qwen2.5-7B) on math, reasoning, and coding tasks
> Coding: Strong performance on HumanEval, MBPP, and BigCodeBench

Reasoning Benchmarks:

> Reasoning-enhanced Phi-4-Mini outperforms DeepSeek-Rl-Distill-Llama-8B and matches DeepSeek-Rl-Distill-Qwen-7B on AIME, MATH-500, and GPQA Diamond

[m_ric @AymericRoucher]

Introducing open-Deep-Research by @huggingface ! 💥

Deep Research from @OpenAI is really good... But it's closed, as usual.

> So with a team of cracked colleagues, we set ourselves a 24hours deadline to replicate and open-source Deep Research!

➡️ We built open-Deep-Research, an entirely open agent that can: navigate the web autonomously, scroll and search through pages, download and manipulate files, run calculation on data...

We aimed for the best performance: are the agent's answers really rigorous?

On GAIA benchmark, Deep Research had 67% accuracy on the validation set.
➡️ open Deep Research is at 55% (powered by o1), but it is:
- the best pass@1 solution submitted
- the best open solution

And it's only getting started ! Please jump in, drop PRs, and let's bring it to the top 🚀

[OpenBMB @OpenBMB]

📄 Finally here! MiniCPM-o 2.6 Technical Blog

🌟 Key Highlights:
🌐 End-to-end Omni-modal Architecture
🎥 Omni-modal Live Streaming Mechanism
🎤 Configurable Speech Modeling Design

🔗 More details: https://t.co/44bPjqLTlC

[Andi Marafioti @andimarafioti]

Introducing the smollest VLMs yet! 🤏
SmolVLM (256M & 500M) runs on <1GB GPU memory.
Fine-tune it on your laptop and run it on your toaster. 🚀
Even the 256M model outperforms our Idefics 80B (Aug '23).
How small can we go? 👀

[Flower @flwrlabs]

Federated Learning Meets Robotics: 🤖 LeRobot + 🌼 Flower

This demo demonstrates how robots in remote environments can collaboratively train an AI model using their local data, which is then aggregated into a shared model.

In this quickstart, you will train a Diffusion policy on the PushT dataset using Flower Datasets for easy partitioning. This combines the power of Flower and the @huggingface @LeRobotHF project! (Cheers to @RemiCadene, @Thom_Wolf, and the entire @LeRobotHF team!)

Special thanks to Flower contributor: Ivelin Ivanov (@ivelini), founder of https://t.co/kXjafN9gJt, Mentor and Venture Advisor at the Founder Institute (@founding), whose expertise made this possible. After inspiring us in the last Flower Monthly, Ivelin contributes yet again, enriching the FL community with this practical resource.

Ready to start?
Find the link in the comments below!

#FederatedLearning #RoboticsAI #MachineLearning #CollaborativeAI #DecentralizedAI

[Hugging Face @huggingface]

Open-source army 🫡🫡🫡