JeffLai

AI/ML Researcher, PhD from UC Davis.

[JeffLai @JeffLaiZF]

🚀📷 Excited to share that we have released VeCLIP to improve CLIP pre-training.

Github: https://t.co/LAxRHBEPIN
ArXiv: https://t.co/z6HBPcoP6R

We achieved 83.1% 0-shot on ImgNet with H14, trained on DFN-5B and 300M synthetic captions.

[Zhiting Hu @ZhitingHu]

🔥Really excited to see the release of PAN world model, a project I had been working over the past years.

PAN is a general world model capable of simulating physical, agentic, and nested worlds, synthesizing infinite interactive experiences for training AI agents.

Building on top of pretrained LLMs and video diffusion models, PAN connects language, perception, action, and latent thoughts, for long-horizon simulation and reasoning.

PAN shows overwhelming performance gains over JEPA-2, Cosmos-2, and other prior models. More in the thread👇 ... 1/

[𝚐𝔪𝟾𝚡𝚡𝟾 @gm8xx8]

SF-LLaVA-1.5 is a family of video LLMs using a two-stream SlowFast design for token-efficient long-form video understanding. Models range from 1B to 7B parameters and are trained with a streamlined pipeline on public data. They perform competitively across video and image benchmarks, achieving SOTA on long-form video tasks like LongVideoBench and MLVU, with strong results even at small scales.

���

[Zongyu (Johnson) Lin @zy27962986]

🚀🚀🚀Want to develop a cutting-edge video generation model towards Sora? Please dive into Apple’s latest recipe and studies for scalable video generation models🔥🔥🔥. In this work, we aim at providing a transparent and detailed recipe 📖 for model architecture, training strategy and data for scalable text-image conditioned video generation.
Arxiv: https://t.co/VDZ2o3X2bz
Hugging Face link: https://t.co/iL8QmdpRtA

[meng shao @shao__meng]

# AI 论文分享
CLOC: 基于局部对比学习的语言-图像预训练方法

论文提出了一种���为 CLOC (Contrastive Localized Language-Image Pre-Training) 的新方法, 旨在改进CLIP 模型的局部化能力��CLIP 是一种广泛使用的视觉-语言预训练模型, 但它主要关注全局图像-文本对齐, 而忽视了区域级别的细粒度理解。

# 论文的主要内容
- 提出了 "可提示嵌入" (Promptable Embeddings) 的概念, 使图像编码器能够根据空间提示 (如边界框) 生成区域表示
- 设计了一个轻量级的 "Prompter" 模块, 可以从 CLIP 的图像嵌入中提取区域嵌入
- 开发了一个大规模的伪标注数据引擎 VESL (Visually-Enriched and Spatially-Localized), 可以生成高质量的区域-文本对
- 在多个下游任务上进行了广泛的实验, 证明了 CLOC 的有效性

# 研究的创新解读
- 首先, 它解��了 CLIP 缺乏局部化能力的问题, 而不是完全重新设计模型。通过引入 Prompter 模块和区域-文本对比损失, CLOC 在保持 CLIP 原有优势的同时, 增强了其对细粒度视觉理解的能力。
- 其次, VESL 数据引擎的设计非常聪明。它利用视觉丰富的图像描述和开放词汇目标检测器, 生成了大规模的高质量区域-文本标注。这解决了缺乏大规模区域-文本数据集的问题。
- 再者, 论文在各种下游任务上进行了全面的评估, 包括零样本区域任务、多模态大语言模型任务等。这充分证明了 CLOC 的通用性和有效性。

# 论文信息
来自 Apple
作者 Hong-You Chen @JeffLaiZF @HaotianZhang4AI @yinfeiy @zhegan4 等研究者
地址 https://t.co/vB5VhS78nU

[Zhe Gan @zhegan4]

🚀🚀 Thrilled to share MM1.5! MM1.5 is a significant upgrade of MM1. With one single set of weights, MM1.5 excels at (1) read your charts, tables, any text-rich images, (2) understand visual prompts like points and boxes, provide grounded outputs, and (3) multi-image reasoning.

🔥🔥 We also introduce two variants: (1) MM1.5-UI to understand your iPhone screen 📱, and (2) MM1.5-Video for video inputs 🎥.

As a research study, we also share the detailed ablations that guided our research process (🧵)

[Zhe Gan @zhegan4]

💡CLIP is the default choice for most multimodal LLM research. But, we know CLIP is not perfect. It is good at high-level semantics, but not for capturing fine-grained info.

🤩🤩 We present CLOC ⏰, our next-generation image encoder, with enhanced localization capabilities, and serves as a drop-in replacement for CLIP.

🚀🚀How to do that? We conduct large-scale pre-training with region-text supervision pseudo-labelled on 2B images.

🎁As a result, CLOC is indeed a better image encoder, not only for zero-shot image/region tasks, but also for multimodal LLM.

[Shuang Ma @shuangma3]

🚀 Excited to announce MMAU benchmark!
🌟 Are you in search of a benchmark for LLM agents? Introducing MMAU, providing comprehensive and multi-dimensional evaluations for LLMs as agents. 🧵👇 1/

Paper: https://t.co/RdiAyQKJLU
Code: https://t.co/XrqnJ1gUM7

[AK @_akhaliq]

Apple presents MMAU

A Holistic Benchmark of Agent Capabilities Across Diverse Domains

Recent advances in large language models (LLMs) have increased the demand for comprehensive benchmarks to evaluate their capabilities as human-like agents. Existing benchmarks, while useful, often focus on specific application scenarios, emphasizing task completion but failing to dissect the underlying skills that drive these outcomes. This lack of granularity makes it difficult to deeply discern where failures stem from. Additionally, setting up these environments requires considerable effort, and issues of unreliability and reproducibility sometimes arise, especially in interactive tasks. To address these limitations, we introduce the Massive Multitask Agent Understanding (MMAU) benchmark, featuring comprehensive offline tasks that eliminate the need for complex environment setups. It evaluates models across five domains, including teal{Tool-use}, teal{Directed Acyclic Graph (DAG) QA}, teal{Data Science and Machine Learning coding}, teal{Contest-level programming} and teal{Mathematics}, and covers five essential capabilities: orange{Understanding}, orange{Reasoning}, orange{Planning}, orange{Problem-solving}, and orange{Self-correction}. With a total of 20 meticulously designed tasks encompassing over 3K distinct prompts, MMAU provides a comprehensive framework for evaluating the strengths and limitations of LLM agents. By testing 18 representative models on MMAU, we provide deep and insightful analyses. Ultimately, MMAU not only sheds light on the capabilities and limitations of LLM agents but also enhances the interpretability of their performance.

[JeffLai @JeffLaiZF]

VeCLIP introduce detailed visual-enriched captions from LLaVA and Vicuna, incorporating the meta information from its original caption.

[JeffLai @JeffLaiZF]

🚀📷 Excited to share that we have released VeCLIP to improve CLIP pre-training.

Github: https://t.co/LAxRHBEPIN
ArXiv: https://t.co/z6HBPcoP6R

We achieved 83.1% 0-shot on ImgNet with H14, trained on DFN-5B and 300M synthetic captions.

[Zhe Gan @zhegan4]

🚀🚀 Excited to release code & ckpt for our new image encoders.

1. VeCLIP: https://t.co/c2xHMRABr4

83.1% 0-shot on ImgNet with H14, trained on DFN-5B and 300M synthetic captions

2. MOFI: https://t.co/bAvAWegqZ8

SOTA on image retrieval, trained on 1B entity-annotated images.

[JeffLai @JeffLaiZF]

@zhegan4 Thanks Zhe, I seldom use Twitter haha

[Zhe Gan @zhegan4]

🌟 Introducing VeCLIP: Improving CLIP training via visual-enriched captions
📘https://t.co/nKguy7avrg
⛽️ Data is the fuel for CLIP training, however, alt-text can be noisy.
🚀🚀 By using Vicuna and LLaVA for text rewriting, VeCLIP boosts CLIP perf across 3M-200M data scales.

[JeffLai @JeffLaiZF]

@NielsRogge @zhegan4 Thanks Niels! These two papers either use LLM alone or captioning model alone. The former cannot add new info while the latter may lack the powerful language ability. We design our pipeline to fuse Captioning + LLM to further boost caption quality.

[JeffLai @JeffLaiZF]

@AmmarHaydari China has not only AI but also 5G