Matthias Minderer

Research Scientist at @GoogleResearch.

[Andreas Steiner @AndreasPSteiner]

Looking for a small or medium sized VLM? PaliGemma 2 spans more than 150x of compute!

Not sure yet if you want to invest the time 🪄finetuning🪄 on your data? Give it a try with our ready-to-use "mix" checkpoints:

🤗 https://t.co/rdVkdRLmEo
🎤 https://t.co/lLsAVkANhI

[Ibrahim Alabdulmohsin | إبراهيم العبدالمحسن @ibomohsin]

🔥Excited to introduce RINS - a technique that boosts model performance by recursively applying early layers during inference without increasing model size or training compute flops! Not only does it significantly improve LMs, but also multimodal systems like SigLIP.
(1/N)

[Alexander Kolesnikov @__kolesnikov__]

I always dreamed of a model that simultaneously

1. optimizes NLL of raw pixel data,
2. generates competitive high-res. natural images,
3. is practical.

But it seemed too good to be true. Until today!

Our new JetFormer model (https://t.co/OXuSnnyuYt) ticks on all of these.

🧵

[Matthias Minderer @MJLM3]

@XiaohuaZhai @OpenAI @giffmana @__kolesnikov__ Congrats to all of you!

[Matthias Minderer @MJLM3]

@unsorsodicorda @_akhaliq Different metrics. It's an increasing problem with LVIS in the literature. There's LVIS AP on the full val set, which generally produces the lowest numbers. That's what we report. Then there's minimal, and also "fixed" AP, both giving higher numbers. Some report those as "LVIS".

[Matthias Minderer @MJLM3]

@unsorsodicorda @_akhaliq @Thom_Wolf Note that this HF benchmark includes the text tower. Image only (with precomputed text embeddings) would be significantly faster.

[Matthias Minderer @MJLM3]

@unsorsodicorda @_akhaliq @Thom_Wolf I haven't benchmarked the HF implementation myself, but this page reports 22.395ms / image on V100 for, I assume, OWLv1 B/32 at res 768, which is very roughly equivalent in compute to B/16 at res 400. So it is at least roughly in the same ballpark: https://t.co/pn1SBkjeaB

[Ibrahim Alabdulmohsin | إبراهيم العبدالمحسن @ibomohsin]

How is next-token prediction capable of such intelligent behavior? I’m very excited to share our work, where we study the fractal structure of language. TLDR: thinking of next-token prediction in language as “word statistics” is a big oversimplification!
https://t.co/h2m9gsisVp

[Matthias Minderer @MJLM3]

@ahatamiz1 @giffmana (1) No, so it would not have been trivial to reproduce, although the pretrained checkpoints I used and fine-tuning code is available.
(2) Maybe the strong size augmentation+mosaics used during OWL-ViT training helps low-res performance? Need to investigate this further.

[Matthias Minderer @MJLM3]

@giffmana @ahatamiz1 The O365+VG-finetuned checkpoints are indeed not available (yet). Happy to work with the authors to make the results reproducible easily.

[Matthias Minderer @MJLM3]

@arankomatsuzaki I added OWL-ViT v2 to the plot. A single OWLv2 B/16, finetuned on O365+VG, covers all speed/accuracy combinations: Simply adjust the inference resolution to match your latency requirements. No re-training needed. https://t.co/eSRZpNYYlW

[Niels Rogge @NielsRogge]

Excited to share that @Google's OWLv2 model is now available in 🤗 Transformers! This model is one of the strongest zero-shot object detection models out there, improving upon OWL-ViT v1 which was released last year🔥

How? By self-training on web-scale data of over 1B examples⬇️

[Thomas Kipf @tkipf]

I'll give a talk on object-centric models for video and 3D at the @ICCVConference Workshop on Large-scale Video Object Segmentation!

Today @ 3:30pm (Room S02)
Website: https://t.co/L5StV6Elh3

I'll cover DORSal (see below) & recent work from our team on structured video models.

[Matthias Minderer @MJLM3]

We just open-sourced OWL-ViT v2, our improved open-vocabulary object detector that uses self-training to reach >40% zero-shot LVIS APr. Check out the paper, code, and pretrained checkpoints: https://t.co/eSRZpNYYlW https://t.co/5f5Sgk5Xfs. With @agritsenko and @neilhoulsby.

[Matthias Minderer @MJLM3]

@neilhoulsby Congrats Neil!

[Matthias Minderer @MJLM3]

@rom1504 @m__dehghani @_basilM @neilhoulsby @JonathanHeek @mcaron31 @AndreasPSteiner @joapuipe @ibomohsin @PiotrPadlewski @MarioLucic_ You could also keep all pixels and add a little bit of padding

[Matthias Minderer @MJLM3]

Check out NaViT, a Vision Transformer that processes images at their native resolution. Apart from improving efficiency and performance of image-level tasks, pretraining at native resolution also produces better backbones for localization tasks like object detection.

[AK @_akhaliq]

Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution

paper page: https://t.co/B8KR2SB99R

The ubiquitous and demonstrably suboptimal choice of resizing images to a fixed resolution before processing them with computer vision models has not yet been successfully challenged. However, models such as the Vision Transformer (ViT) offer flexible sequence-based modeling, and hence varying input sequence lengths. We take advantage of this with NaViT (Native Resolution ViT) which uses sequence packing during training to process inputs of arbitrary resolutions and aspect ratios. Alongside flexible model usage, we demonstrate improved training efficiency for large-scale supervised and contrastive image-text pretraining. NaViT can be efficiently transferred to standard tasks such as image and video classification, object detection, and semantic segmentation and leads to improved results on robustness and fairness benchmarks. At inference time, the input resolution flexibility can be used to smoothly navigate the test-time cost-performance trade-off. We believe that NaViT marks a departure from the standard, CNN-designed, input and modelling pipeline used by most computer vision models, and represents a promising direction for ViTs.

[AK @_akhaliq]

Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution

paper page: https://t.co/B8KR2SB99R

The ubiquitous and demonstrably suboptimal choice of resizing images to a fixed resolution before processing them with computer vision models has not yet been successfully challenged. However, models such as the Vision Transformer (ViT) offer flexible sequence-based modeling, and hence varying input sequence lengths. We take advantage of this with NaViT (Native Resolution ViT) which uses sequence packing during training to process inputs of arbitrary resolutions and aspect ratios. Alongside flexible model usage, we demonstrate improved training efficiency for large-scale supervised and contrastive image-text pretraining. NaViT can be efficiently transferred to standard tasks such as image and video classification, object detection, and semantic segmentation and leads to improved results on robustness and fairness benchmarks. At inference time, the input resolution flexibility can be used to smoothly navigate the test-time cost-performance trade-off. We believe that NaViT marks a departure from the standard, CNN-designed, input and modelling pipeline used by most computer vision models, and represents a promising direction for ViTs.

[Aran Komatsuzaki @arankomatsuzaki]

Scaling Open-Vocabulary Object Detection

Proposes OWLv2, which achieves SotA open-vocabulary detection already at 10M examples and further large improvement by scaling to over 1B examples.

https://t.co/8JXQecS97j