@xxxnell @_akhaliq Thanks Namuk for the great analysis. Please have a look at my paper “SiT: self-supervised vision transformers” one of the early papers that leverage self-supervised learning for vision transformers, proposing MIM with vision transformers and combining it with contrastive learning
@ylecun@lexfridman True, but we managed to make it work with high performance on small datasets. Unfortunately the paper got rejected because we didn’t train on ImageNet which is not feasible for us as a small group.
#KostasThoughts#CVPR2022: There is no shame getting the initial review wrong, e.g., missing an important detail. It is wrong when you let your ego get into the way of arriving to the correct decision.
@y_m_asano You are right. But the GMML is not just for augmentation, but rather extract the most information from the available data which made it feasible to obtain high accuracy on small datasets when trained from scratch. I think even with more data, GMML will be beneficial.
@y_m_asano and this masking encourages the network to learn from semantic information present in the locality of the masked tokens to recover the missing information as well as the uncorrupted parts, and hence, the network will unlikely to overfit.
2/2
@y_m_asano I think the situation is different in vision, but also it depends on how the network is trained. Unlike NLP, in SiT, we apply severe corruption on a [group] of connected patches
1/2
@y_m_asano You can check “SiT: self supervised vision transformer”. It’s the same idea, proposed long ago, and because we don’t have enough compute, we only show the results on small datasets where we outperform the SOTA at that time.
@francoisfleuret@ak92501@Meta You are right. I was really disappointed at that time, and to be honest, I did not expect that anyone will pay attention to my respond as I am not even a twitter user.