Denise Welsch

Teaching and developing in math/stats/ML/Bioinf. Interested in humans.

[Sundar Pichai @sundarpichai]

Very happy to support this on behalf of Google. We have long benefited from open source, are big contributors to open source and in fact have consistently made open weights models with Gemma available from @GoogleDeepMind @demishassabis . Onwards!

[Haiyu Wu @HaiyuWu1]

Recent discussions about open-sourcing make me feel that I should go back and revisit these important open-source works in representation learning that pushed the field forward and eventually made vision foundation models so useful in applications.

These are the papers that come to mind. Feel free to correct my opinions and share any important papers that I may have missed.

MoCo v1 (2019-11) – ResNet
Method: contrastive loss + a queue of samples from previous batches for negative pairing + EMA-based weight updates
# of training branches: 2
Effect: EMA-based weight updates became an important training paradigm in later representation learning algorithms, but this work used EMA only to produce stable embeddings for negative samples.

SimCLR (2020-02) – ResNet
Method: contrastive loss + more augmentations + an MLP projector to capture the effects of augmentations
# of training branches: 2
Effect: An MLP projector became the gold standard in representation learning algorithms.

BYOL (2020-06) – ResNet
Method: positive pairs only + an asymmetric architecture + stop-gradient + SimCLR augmentations + EMA
# of training branches: 2
Effect: Probably the first representation learning method without negative pairs. It inspired DINO and probably JEPA in terms of architectural design.

SwAV (2020-06) – ResNet
Method: global/local views + SimCLR augmentations + class prototypes
# of training branches: 2
Effect: Its global/local-view training became the standard setting in later representation learning methods.

SimSiam (2020-11) – ResNet
Method: BYOL without EMA + a constant learning rate for the predictor
# of training branches: 2

Barlow Twins (2021-03) – ResNet
Method: global views + SimCLR augmentations + batch-level covariance optimization
# of training branches: 2
Effect: The optimization target changed. Previous works used cosine similarity, but this work uses covariance, which explicitly regularizes the latent space.

MoCo v3 (2021-04) – ViT
Method: MoCo v1 - memory bank + SimCLR augmentations and an MLP + a frozen first layer
# of training branches: 2
Effect: The first work to show that ViT can be used for representation learning.

DINO (2021-04) – ViT
Method: BYOL - the predictor + global/local views + softmax distillation + centering + temperature sharpening
# of training branches: 2
Effect: Enabled the era of large-scale vision foundation models.

VICReg (2021-05) – ViT
Method: Barlow Twins → branch-level covariance + variance + sample-level invariance
# of training branches: 2 or 1
Effect: The first representation learning method to support stable one-branch training without EMA, centering, layer freezing, temperature sharpening, or stop-gradient.

MAE (2021-11) – ViT
Method: learning representations through pixel reconstruction
# of training branches: 1
Effect: A simple, simple, and simple training strategy that is widely used in medical imaging, from what I have heard.

I-JEPA (2023-01) – ViT
Method: learning representations through latent prediction
# of training branches: 2
Effect: The first representation learning method in the spirit of JEPA. It is also the basis of V-JEPAs, VL-JEPA, and so on.

DINOv2 (2023-04) – ViT
Method: DINO v1 + SK centering + KoLeo + LayerScale
# of training branches: 2
Effect: The first large vision foundation model to show superior performance across a wide range of downstream tasks.

DINOv3 (2025-08) – ViT
Method: DINO v2 + Gram anchoring + flattened schedules
# of training branches: 2
Effect: The first large vision foundation model to show superior performance on fine-grained tasks, such as segmentation and depth estimation, and to achieve accuracy comparable to supervised learning methods on ImageNet-1K.

LeJEPA (2025-11) – ViT + others
Method: VICReg → full-distribution regularization. It provides mathematical proofs of why an isotropic distribution is the ideal target for the latent space and proposes SIGReg, a method that is linear in the scaling factors.
# of training branches: 1
Effect: No heuristics are needed for training stability. It provides a simple solution for various tasks, including world models, such as the LeWM series; multimodal learning, such as LeVLJEPA; drone applications, such as SkyJEPA; and so on.

VISReg (2026-06) – ViT
Method: SIGReg → a faster and stronger anti-collapse method that can form an isotropic Gaussian distribution more efficiently. It decomposes the regularization into center, variance, and shape losses, allowing the model to begin introducing isotropy at an early stage of training.
# of training branches: 1
Effect: It is too new to summarize concretely.

[Denise Welsch @statsomat]

I'm happy to announce that our project GLUH (Gene Prediction by Leveraging Unaligned Locus Homology Information) has received funding from the Carl-Zeiss Foundation https://t.co/yYD3ghZ8MS

[Denise Welsch @statsomat]

Our Master's students at Hochschule Koblenz just wrapped up the https://t.co/cAwXhpaxYT app, a web tool for automated PCA, built with learners in mind. It's part of our Statsomat teaching concept: GitHub, interpretation in natural language, open ed for the global community.

[Denise Welsch @statsomat]

@Goemario Impressive x 2.

[Mario Stanke @Goemario]

This AI-generated popular science podcast by NotebookLM about our Tiberius preprint would likely not have been done so good by human journalists.
https://t.co/QGehWRHSZT

[Hitarth Sharma @iamhitarth]

Mind keeps getting blown every time I see this comparison between @OpenAI GPT-4, @AnthropicAI Claude Opus and @Meta Llama 3 70B on @GroqInc in a post I'm putting together...

~17x lower input cost 🤯

~38x lower output cost 🤯🤯

14x faster @ ~280 vs ~20 tokens per sec 🤯🤯🤯

[Yann LeCun @ylecun]

Effective altruism: limulus version.
Doesn't require too many neurons, apparently.

[Yann LeCun @ylecun]

Can mathematical obfuscation help to get papers accepted?

[Yann LeCun @ylecun]

Trajectory planning with 800 million neurons.

[Denise Welsch @statsomat]

The Levenshtein distance, explained in a lecture at the University of Applied Sciences Würzburg-Schweinfurt. In German.
https://t.co/REj79IPuyf

[Denise Welsch @statsomat]

A well known experiment and the consequences for the Chi² Test: The Lady Tea Testing experiment of Sir R. Fisher. Check the slides from a talk: https://t.co/pETe3a9F1D

[Denise Welsch @statsomat]

I am thrilled that the Statsomat apps https://t.co/8rUQrAvtsD have been selected to move to phase II of The Learning Engineering Tools Competition https://t.co/fX3UhVXIbn

[Denise Welsch @statsomat]

You can install now the Statsomat apps as the Statsomat R package from CRAN https://t.co/DIWxMTEdqd

[Denise Welsch @statsomat]

The Statsomat Apps with R and Python https://t.co/9z0ozFc257 via @Rbloggers

[Denise Welsch @statsomat]

Particularly proud to announce the launching of the app for Multiple Comparisons Procedures with a Control (Dunnet) in an experimental beta version. Check the app: https://t.co/s2vKYxcFHB.

[Denise Welsch @statsomat]

For a technical background of the Statsomat apps, check out the webinar: https://t.co/Jmh4C0JbDm

[François Chollet @fchollet]

Most people I meet overestimate what deep learning can do (it's curve-fitting, don't expect it to do discrete symbol manipulation, it will solve symbolic tasks via embedding + interpolation) and simultaneously underestimate what you can do with curve-fitting given enough data

[Denise Welsch @statsomat]

Please consider the next talk about the open-source Statsomat apps: https://t.co/nxXW3AiVYy

[Denise Welsch @statsomat]

The Statsomat/EDAPY app for Exploratory Data Analysis (EDA) with Python is finally online. Check it out: https://t.co/esaOfz9tNc