Shen Yan

[AK @_akhaliq]

Bytedance just dropped Seed1.5-VL on Hugging Face

Achieves top performance with a relatively modest architecture, 532M vision encoder & 20B active parameter MoE LLM.

Delivers State-of-the-Art results on 38 out of 60 public VLM benchmarks, demonstrating broad competence.

[AK @_akhaliq]

Google announces Streaming Dense Video Captioning

An ideal model for dense video captioning -- predicting captions localized temporally in a video -- should be able to handle long input videos, predict rich, detailed textual descriptions, and be able to produce outputs

[AK @_akhaliq]

Google announces VideoPrism

A Foundational Visual Encoder for Video Understanding

introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. We pretrain VideoPrism on a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding by global-local distillation of semantic video embeddings and a token shuffling scheme, enabling VideoPrism to focus primarily on the video modality while leveraging the invaluable text associated with videos. We extensively test VideoPrism on four broad groups of video understanding tasks, from web video question answering to CV for science, achieving state-of-the-art performance on 30 out of 33 video understanding benchmarks.

[AK @_akhaliq]

Google presents PaLM2-VAdapter

Progressively Aligned Language Model Makes a Strong Vision-language Adapter

paper demonstrates that a progressively aligned language model can effectively bridge frozen vision encoders and large language models (LLMs). While the fundamental architecture and pre-training methods of vision encoders and LLMs have been extensively studied, the architecture and training strategy of vision-language adapters vary significantly across recent works. Our research undertakes a thorough exploration of the state-of-the-art perceiver resampler architecture and builds a strong baseline. However, we observe that the vision-language alignment with perceiver resampler exhibits slow convergence and limited scalability with a lack of direct supervision. To address this issue, we propose PaLM2-VAdapter, employing a progressively aligned language model as the vision-language adapter. Compared to the strong baseline with perceiver resampler, our method empirically shows faster convergence, higher performance, and stronger scalability. Extensive experiments across various Visual Question Answering (VQA) and captioning tasks on both images and videos demonstrate that our model exhibits state-of-the-art visual understanding and multi-modal reasoning capabilities. Notably, our method achieves these advancements with 30~70% fewer parameters than the state-of-the-art large vision-language models, marking a significant efficiency improvement.

[AK @_akhaliq]

Google announces Pixel Aligned Language Models

paper page: https://t.co/BeN1JFJOU7

Large language models have achieved great success in recent years, so as their variants in vision. Existing vision-language models can describe images in natural languages, answer visual-related questions, or perform complex reasoning about the image. However, it is yet unclear how localization tasks, such as word grounding or referring localization, can be performed using large language models. In this work, we aim to develop a vision-language model that can take locations, for example, a set of points or boxes, as either inputs or outputs. When taking locations as inputs, the model performs location-conditioned captioning, which generates captions for the indicated object or region. When generating locations as outputs, our model regresses pixel coordinates for each output word generated by the language model, and thus performs dense word grounding. Our model is pre-trained on the Localized Narrative dataset, which contains pixel-word-aligned captioning from human attention. We show our model can be applied to various location-aware vision-language tasks, including referring localization, location-conditioned captioning, and dense object captioning, archiving state-of-the-art performance on RefCOCO and Visual Genome.

[AK @_akhaliq]

Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners
abs: https://t.co/pP04wT0f6Q

[AK @_akhaliq]

Multiview Transformers for Video Recognition
abs: https://t.co/Qh5tOu6VHd

[Shen Yan @shen_82]

In this work, we develop a method to create surrogate NAS benchmarks that include full training information for each architecture, including train, validation, and test loss and accuracy learning curves.

[Shen Yan @shen_82]

I will present our paper on creating multi-fidelity NAS benchmarks and the potential of using learning curve prediction algorithms today in #NeurIPS2021 from 4:30 to 6pm PT. Stop by to ask questions.

Poster: https://t.co/aZS7NqzVW0
Paper & Code: https://t.co/2iXaQDxb1g

[Shen Yan @shen_82]

I'll present our work on computation-aware architecture encoding at a #ICML21 poster session today (Tuesday) 9pm - 11pm PDT. Stop by to ask questions and find out more.

Long talk & poster: https://t.co/OPytyhcrD3
Paper: https://t.co/aHBB3L91oT
Code: https://t.co/HoIh2HcYKR

[Gabriel Peyré @gabrielpeyre]

Floyd–Warshall algorithm computes all pairs shortest distances. Related to iterative powering of the adjacency matrix for the (min,+) algebra from Tropical geometry. https://t.co/CrAgWDka0H https://t.co/59bbMAqmkT https://t.co/gvKj2Y7FK8

[Shen Yan @shen_82]

@hardmaru @elonmusk Looks like regularized evolution is all you need

[Shen Yan @shen_82]

on NASBench201 search space:

[Shen Yan @shen_82]

Tonight 12/10 at 9pm PT we will present Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? @NeurIPSConf. You are welcome to discuss!

Paper: https://t.co/ExDIvw3Yhf
Pres: https://t.co/W5gfNGmtop

[Shen Yan @shen_82]

We visualize the decoded architecture cells from the learned latent space and found pretrain can better group similar connections and operations together. e.g. on NASBench101 search space, top: pretrain, bottom: supervised.

[Shen Yan @shen_82]

Nice blog. EM is indeed one of my favorite optimization algorithms.

[Karin Knudson @karinknudson]

Blog post: Expectation Maximization (EM)

https://t.co/oVCSCDqFyW