Keyu Tian

Master's student @ Peking University. Incoming Ph.D. student.

[Keyu Tian @keyutian]

@noahcjones here is a brief comparison

[Keyu Tian @keyutian]

Now this paper has been accepted to #ICLR2023 as a spotlight🌟! I truly appreciate everyone's efforts. Check out our repo https://t.co/zjGbmLtZYI for the latest demo video and updates! Title: "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"

[Sebastian Raschka @rasbt]

How can we leverage successful pretraining techniques from transformers to improve purely convolutional networks? The answer is *Sparse Convolutions*!

Let's see what happens when purely convolutional networks are pretrained with 1.28 million unlabeled images ...

1/7

[Python Trending 🇺🇦 @pythontrending]

SparK - [ICLR'23 Spotlight] The first successful BERT-style pretraining on any *convolutional network*; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling" https://t.co/vdgxhmPxS1

[/MachineLearning @slashML]

: The first BERT-style pretraining on CNNs! #nlp #cnn https://t.co/L7GT2MFTNH

[Keyu Tian @keyutian]

@foozlefoo Thanks, we're glad SparK may be helpful to your project! Sorry we delayed the update for a few days because it's Lunar New Year today, we'll get back to you as soon as they're posted.

[Keyu Tian @keyutian]

PS: our original submission in Oct. 2022 was titled "Sparse and Hierarchical Masked Modeling for Convolutional Representation Learning"; we've changed it to the new one to more clearly express our vision for bringing the powerful BERT-style self-supervised learning to CNNs.

[Keyu Tian @keyutian]

@rasbt Thank you so much! Your notes are really helpful in letting people know about what we do! And I'm glad to see this work has the opportunity to make a meaningful contribution to our community.

[Daisuke Okanohara / 岡野原 大輔 @hillbig]

画像認識の事前学習で、ViTは一部の入力をMaskし、それを予測するタスクが成功していたがConvは隣り合うパッチ間で重複があり情報が漏れ成功してなかった。SparKは出力も入力と同じマスクを維持するSparseConvを使うことで問題を解決、また復号器側も階層を持ちさらに改善 https://t.co/PWkAoOQXTl

[Keyu Tian @keyutian]

@reidatcheson @rasbt Thanks for your interest! We are writing a google Colab tutorial to make it easy to play with our pre-trained models (e.g. to reconstruct the images you upload). We'll release it before this weekend and hope it can be helpful.

[Keyu Tian @keyutian]

@rasbt Yeah, that's true for point 2. It'd be better to say "CNNs achieved higher performance than Transformers".

BTW, I really appreciate the contrastive idea, which seems more natural for image data than randomly masking. But it turns out the BERT-style pretraining is more effective.

[Sebastian Raschka @rasbt]

How can we leverage successful pretraining techniques from transformers to improve purely convolutional networks? The answer is *Sparse Convolutions*!

Let's see what happens when purely convolutional networks are pretrained with 1.28 million unlabeled images ...

1/7

[fly51fly @fly51fly]

[CV] Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling
K Tian, Y Jiang, Q Diao, C Lin, L Wang, Z Yuan [Peking University & Bytedance Inc & University of Oxford] (2023)
https://t.co/TJwQep1Has
#MachineLearning #ML #AI #CV
[1/2]

[Keyu Tian @keyutian]

@HappyToKnowThat @jamesr66a it's kinda like the number of neurons in a brain, the more, the smarter. And each parameter is a float in Java, so you can imagine how much memory it takes.

[Keyu Tian @keyutian]

@ducha_aiki Thanks for sharing😃! For those who may be interested in the code implementation, please see: https://github[dot]com/keyu-tian/SparK

[Keyu Tian @keyutian]

@Deep__AI Thanks for sharing😃! For those who may be interested in the code implementation, please see: https://github[dot]com/keyu-tian/SparK

[DeepAI @DeepAI]

🤩The first successful BERT-style #SelfSupervisedLearning on any convolutional network! #ResNet now enjoys masked autoencoding! 🚀A breakthrough paper "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling" by @keyutian et al.
https://t.co/YmnyCsOKHj

[Dmytro Mishkin 🇺🇦 @ducha_aiki]

Designing BERT for convolutional networks: sparse and hierarchical masked modeling

Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, Zehuan Yuan

tl;dr: create image, which looks to CNN same, as transformers -> MIM starts working
https://t.co/g6zDN0BbEH

[Keyu Tian @keyutian]

@kzmolikova @rasbt Thanks for noticing this, that is our work which's been on OpenReview since Oct. 2022 (https://openreview. net/forum?id=NRxydtWup1S), and uploaded to arxiv recently.

It gives some different results from ConvNeXt v2, e.g., our method works pretty well on ConvNeXt v1 and ResNet.

[Keyu Tian @keyutian]

@DiChang10 @eccvconf cool, hope to see you on next ECCV!