Frederik Hvilshøj

ML Lead @ Encord

[Frederik Hvilshøj @FHvilshoj]

Here are some useful resources
1️⃣ A technical blog post: https://t.co/M4HtR3KYfr
2️⃣ My 5-minute explainer: https://t.co/5x8xxwqNfu

... and by the way,

> We have also integrated SAM 2 into our labeling platform.

[Frederik Hvilshøj @FHvilshoj]

Last night, Meta AI announced the release of Segment Anything 2 (SAM 2) 🤯
The implications for AI teams are HUGE. I’ve taken a deep dive into SAM 2 and explored:

Practical implications
Quantitative implications
The technical masterstrokes
A big new open dataset

[Frederik Hvilshøj @FHvilshoj]

4/ A big new open dataset

A new dataset is introduced

1️⃣ Employs an ingenious data engine taking frame annotations from 38 to 5 secs/frame

2️⃣ Contains +35M masks (50x the previously largest dataset)

3️⃣ Contains +50K videos (11x the previously largest dataset)

[Frederik Hvilshøj @FHvilshoj]

Here are some useful resources

1️⃣ A technical blog post: https://t.co/M4HtR3KYfr
2️⃣ My 5-minute explainer: https://t.co/5x8xxwqNfu

and by the way...

> We have integrated SAM 2 into our labeling platform

[Frederik Hvilshøj @FHvilshoj]

3/ The technical masterstrokes

1️⃣ A new memory mechanism (bank + encoder) gives SAM 2 the tracking ability

2️⃣ An occlusion detector helps remove false positives across frames

3️⃣ A hierarchical image encoder (Hiera) that allows skip connections improve accuracy

[Frederik Hvilshøj @FHvilshoj]

@alishbaimran_ 👏🏻

[Frederik Hvilshøj @FHvilshoj]

You can read the paper here: https://t.co/zdiJBGxhKn
Learn more about DINO here: https://t.co/1QNNGSZNnI
(9/9)

[Frederik Hvilshøj @FHvilshoj]

ICLR'24 is wrapping up. VAEs got the test-of-time award and an outstanding paper award was given to an interesting successor to DINO specifically tailored for videos. I put some more words on it here. (1/9)

[Frederik Hvilshøj @FHvilshoj]

and what's the caveats? (8/9)

[Frederik Hvilshøj @FHvilshoj]

So what's the implications? (7/9)

[Frederik Hvilshøj @FHvilshoj]

The conceptual idea of using temporal information by applying cross-attention to mask the input to the student network intelligently (6/9)

[Frederik Hvilshøj @FHvilshoj]

So why that title? (5/9)

[Frederik Hvilshøj @FHvilshoj]

DORA allows you to compute reusable embeddings (4/9)

[Frederik Hvilshøj @FHvilshoj]

Can you do better than just applying DINO on your data? (3/9)

[Frederik Hvilshøj @FHvilshoj]

Imagine you're the ML lead at a company with many videos but no labels (2/9)

[Frederik Hvilshøj @FHvilshoj]

(3/3) Excited to try it!
Here's link to an interactive demo https://t.co/8ur5VzbYNr
Webinar is here: https://t.co/vu1LalvAYM

[Frederik Hvilshøj @FHvilshoj]

Did you notice the huge step forward for MM LLMs that happened this week?

[Frederik Hvilshøj @FHvilshoj]

(2/3) Three bottlenecks of pre-existing open source models