Wenlong Wu

@waylongox

Explore different kinds of possibilities

Missouri, USA

Joined March 2016

285 Following

21 Followers

37 Posts

waylongox retweeted

Haiyu Wu

@HaiyuWu1

4 days ago

Recent discussions about open-sourcing make me feel that I should go back and revisit these important open-source works in representation learning that pushed the field forward and eventually made vision foundation models so useful in applications. These are the papers that come to mind. Feel free to correct my opinions and share any important papers that I may have missed. MoCo v1 (2019-11) – ResNet Method: contrastive loss + a queue of samples from previous batches for negative pairing + EMA-based weight updates # of training branches: 2 Effect: EMA-based weight updates became an important training paradigm in later representation learning algorithms, but this work used EMA only to produce stable embeddings for negative samples. SimCLR (2020-02) – ResNet Method: contrastive loss + more augmentations + an MLP projector to capture the effects of augmentations # of training branches: 2 Effect: An MLP projector became the gold standard in representation learning algorithms. BYOL (2020-06) – ResNet Method: positive pairs only + an asymmetric architecture + stop-gradient + SimCLR augmentations + EMA # of training branches: 2 Effect: Probably the first representation learning method without negative pairs. It inspired DINO and probably JEPA in terms of architectural design. SwAV (2020-06) – ResNet Method: global/local views + SimCLR augmentations + class prototypes # of training branches: 2 Effect: Its global/local-view training became the standard setting in later representation learning methods. SimSiam (2020-11) – ResNet Method: BYOL without EMA + a constant learning rate for the predictor # of training branches: 2 Barlow Twins (2021-03) – ResNet Method: global views + SimCLR augmentations + batch-level covariance optimization # of training branches: 2 Effect: The optimization target changed. Previous works used cosine similarity, but this work uses covariance, which explicitly regularizes the latent space. MoCo v3 (2021-04) – ViT Method: MoCo v1 - memory bank + SimCLR augmentations and an MLP + a frozen first layer # of training branches: 2 Effect: The first work to show that ViT can be used for representation learning. DINO (2021-04) – ViT Method: BYOL - the predictor + global/local views + softmax distillation + centering + temperature sharpening # of training branches: 2 Effect: Enabled the era of large-scale vision foundation models. VICReg (2021-05) – ViT Method: Barlow Twins → branch-level covariance + variance + sample-level invariance # of training branches: 2 or 1 Effect: The first representation learning method to support stable one-branch training without EMA, centering, layer freezing, temperature sharpening, or stop-gradient. MAE (2021-11) – ViT Method: learning representations through pixel reconstruction # of training branches: 1 Effect: A simple, simple, and simple training strategy that is widely used in medical imaging, from what I have heard. I-JEPA (2023-01) – ViT Method: learning representations through latent prediction # of training branches: 2 Effect: The first representation learning method in the spirit of JEPA. It is also the basis of V-JEPAs, VL-JEPA, and so on. DINOv2 (2023-04) – ViT Method: DINO v1 + SK centering + KoLeo + LayerScale # of training branches: 2 Effect: The first large vision foundation model to show superior performance across a wide range of downstream tasks. DINOv3 (2025-08) – ViT Method: DINO v2 + Gram anchoring + flattened schedules # of training branches: 2 Effect: The first large vision foundation model to show superior performance on fine-grained tasks, such as segmentation and depth estimation, and to achieve accuracy comparable to supervised learning methods on ImageNet-1K. LeJEPA (2025-11) – ViT + others Method: VICReg → full-distribution regularization. It provides mathematical proofs of why an isotropic distribution is the ideal target for the latent space and proposes SIGReg, a method that is linear in the scaling factors. # of training branches: 1 Effect: No heuristics are needed for training stability. It provides a simple solution for various tasks, including world models, such as the LeWM series; multimodal learning, such as LeVLJEPA; drone applications, such as SkyJEPA; and so on. VISReg (2026-06) – ViT Method: SIGReg → a faster and stronger anti-collapse method that can form an isotropic Gaussian distribution more efficiently. It decomposes the regularization into center, variance, and shape losses, allowing the model to begin introducing isotropy at an early stage of training. # of training branches: 1 Effect: It is too new to summarize concretely.

HaiyuWu1's tweet photo. Recent discussions about open-sourcing make me feel that I should go back and revisit these important open-source works in representation learning that pushed the field forward and eventually made vision foundation models so useful in applications.

These are the papers that come to mind. Feel free to correct my opinions and share any important papers that I may have missed.

MoCo v1 (2019-11) – ResNet
Method: contrastive loss + a queue of samples from previous batches for negative pairing + EMA-based weight updates
# of training branches: 2
Effect: EMA-based weight updates became an important training paradigm in later representation learning algorithms, but this work used EMA only to produce stable embeddings for negative samples.

SimCLR (2020-02) – ResNet
Method: contrastive loss + more augmentations + an MLP projector to capture the effects of augmentations
# of training branches: 2
Effect: An MLP projector became the gold standard in representation learning algorithms.

BYOL (2020-06) – ResNet
Method: positive pairs only + an asymmetric architecture + stop-gradient + SimCLR augmentations + EMA
# of training branches: 2
Effect: Probably the first representation learning method without negative pairs. It inspired DINO and probably JEPA in terms of architectural design.

SwAV (2020-06) – ResNet
Method: global/local views + SimCLR augmentations + class prototypes
# of training branches: 2
Effect: Its global/local-view training became the standard setting in later representation learning methods.

SimSiam (2020-11) – ResNet
Method: BYOL without EMA + a constant learning rate for the predictor
# of training branches: 2

Barlow Twins (2021-03) – ResNet
Method: global views + SimCLR augmentations + batch-level covariance optimization
# of training branches: 2
Effect: The optimization target changed. Previous works used cosine similarity, but this work uses covariance, which explicitly regularizes the latent space.

MoCo v3 (2021-04) – ViT
Method: MoCo v1 - memory bank + SimCLR augmentations and an MLP + a frozen first layer
# of training branches: 2
Effect: The first work to show that ViT can be used for representation learning.

DINO (2021-04) – ViT
Method: BYOL - the predictor + global/local views + softmax distillation + centering + temperature sharpening
# of training branches: 2
Effect: Enabled the era of large-scale vision foundation models.

VICReg (2021-05) – ViT
Method: Barlow Twins → branch-level covariance + variance + sample-level invariance
# of training branches: 2 or 1
Effect: The first representation learning method to support stable one-branch training without EMA, centering, layer freezing, temperature sharpening, or stop-gradient.

MAE (2021-11) – ViT
Method: learning representations through pixel reconstruction
# of training branches: 1
Effect: A simple, simple, and simple training strategy that is widely used in medical imaging, from what I have heard.

I-JEPA (2023-01) – ViT
Method: learning representations through latent prediction
# of training branches: 2
Effect: The first representation learning method in the spirit of JEPA. It is also the basis of V-JEPAs, VL-JEPA, and so on.

DINOv2 (2023-04) – ViT
Method: DINO v1 + SK centering + KoLeo + LayerScale
# of training branches: 2
Effect: The first large vision foundation model to show superior performance across a wide range of downstream tasks.

DINOv3 (2025-08) – ViT
Method: DINO v2 + Gram anchoring + flattened schedules
# of training branches: 2
Effect: The first large vision foundation model to show superior performance on fine-grained tasks, such as segmentation and depth estimation, and to achieve accuracy comparable to supervised learning methods on ImageNet-1K.

LeJEPA (2025-11) – ViT + others
Method: VICReg → full-distribution regularization. It provides mathematical proofs of why an isotropic distribution is the ideal target for the latent space and proposes SIGReg, a method that is linear in the scaling factors.
# of training branches: 1
Effect: No heuristics are needed for training stability. It provides a simple solution for various tasks, including world models, such as the LeWM series; multimodal learning, such as LeVLJEPA; drone applications, such as SkyJEPA; and so on.

VISReg (2026-06) – ViT
Method: SIGReg → a faster and stronger anti-collapse method that can form an isotropic Gaussian distribution more efficiently. It decomposes the regularization into center, variance, and shape losses, allowing the model to begin introducing isotropy at an early stage of training.
# of training branches: 1
Effect: It is too new to summarize concretely.

307

339

42K

waylongox retweeted

ai maxxer

@agentsmaxxing

18 days ago

Anthropic just released a free course on loop engineering with Fable 5 00:00 - how Claude Code works under the hood 05:01 - the agentic loop explained 16:21 - the function that 99% of devs ignore 19:01 - why voice beats writing 32:34 - automatic code review with draft PRs 58:39 - Fable 5 for work that's not code This free course replaces any paid Claude Code tutorial. Bookmark it for later.

775

145

114K

waylongox retweeted

Yanhua

@yanhua1010

29 days ago

https://t.co/JpG8zZNEDF

425

812

86K

Wenlong Wu @waylongox

2 months ago

1,010 wearable health signals → 42 that matter. Built /wtf to cut through FDA updates, arXiv papers, SDK changes & trials from Apple/Oura/Garmin/WHOOP — weekly. Open-source ↓ https://t.co/UjVznfqXkD #Wearables #DigitalHealth #HealthAI

waylongox's tweet photo. 1,010 wearable health signals → 42 that matter.

Built /wtf to cut through FDA updates, arXiv papers, SDK changes & trials from Apple/Oura/Garmin/WHOOP — weekly.

Open-source ↓
https://t.co/UjVznfqXkD

#Wearables #DigitalHealth #HealthAI https://t.co/gI5R4Re9tl

waylongox retweeted

向阳乔木

@vista8

3 months ago

如果你每周时间有限，但又想获得前沿AI信息？除了刷推特，分享三个精选信息源，读完基本不落伍。 1. 老牌AI Newsletter 沉浸式翻译原作者Owen推荐我，三年以来更新稳定，每周AI热点和AI工具非常全面。 https://t.co/qWwL2dra7E 2. AK大神去Huggingface后搞的热门AI论文Digg榜。靠人工投票筛选出每日、每周、每月最热AI论文。 https://t.co/WHQdFuoIiv 3. 著名在线阅读器 Readwise 精选每周必读。随着AI火热，AI相关文章越来越多。除AI，还有行业大牛文章推荐，好书推荐，让你不局限于信息茧房，最新一期是138，每周修改URL+1访问即可。 https://t.co/yzQneznWaV

833

193

87K

waylongox retweeted

Lance Martin

@RLanceMartin

5 months ago

https://t.co/IxiG7MDaus

128

879K

waylongox retweeted

Nathan Lambert

@natolambert

4 months ago

Excited to launch the accompanying free RLHF Course for my book. To kick it off, I've released: - Welcome video - Lecture 1: Overview of RLHF & Post-training - Lecture 2: IFT, Reward Models, Rejection Sampling - Lecture 3: RL Math - Lecture 4: RL Implementation I'm going to add question & answer videos throughout the lecture to go deeper on topics that need it, and potentially cover some topics that are too recent and in flux to go in print. I expect 10-15 videos in total over the next few months. At the same time, development around the code for the book is picking up. It's a great time to build the foundation for post-training methods. YT playlist and course landing page below.

natolambert's tweet photo. Excited to launch the accompanying free RLHF Course for my book. To kick it off, I've released:

- Welcome video
- Lecture 1: Overview of RLHF & Post-training
- Lecture 2: IFT, Reward Models, Rejection Sampling
- Lecture 3: RL Math
- Lecture 4: RL Implementation

I'm going to add question & answer videos throughout the lecture to go deeper on topics that need it, and potentially cover some topics that are too recent and in flux to go in print. I expect 10-15 videos in total over the next few months.

At the same time, development around the code for the book is picking up. It's a great time to build the foundation for post-training methods.

YT playlist and course landing page below.

239

194K

waylongox retweeted

小八

@IceBearMiner

4 months ago

https://t.co/XYIFdgF6TL

146

268

44K

waylongox retweeted

实践哥MinLi

@MinLiBuilds

4 months ago

https://t.co/lx2bdjgH5E

321

646K

waylongox retweeted

AYi

@AYi_AInotes

4 months ago

翻墙出来一定要多看优质的信息源，不要只是出来看片！多逛英推区宝藏，看不懂听不懂阿绎给你翻译拆解系列03：这个创造了Claude Code的男人Boris Cherny大神，完整公开了自己的工作流，并直播演示了一半的编码工作在手机上完成🤪 不是回消息，是同时跑5到10个Claude实例，用手机启动任务，去喝咖啡，回来代码写好了🤣 他刚公开了自己的完整工作流，看完之后我发现，核心就三件事，但每一件都跟大多数人的直觉相反：第一，永远选最贵最聪明的模型听着像烧钱，实际上反过来，聪明模型一次想清楚，笨模型来回试错烧掉的token远超差价。他的原话："计划做得好，代码自然好。" 第二，整个团队维护一个纯文本知识库不是Notion，不是花哨的文档系统，就是一个txt文件。Claude每犯一次错，记一笔，每周更新好几次，这个文件就是团队的长期记忆，Claude不会在同一个地方摔倒两次。第三，永远让Claude看到自己代码的运行结果能跑代码，能看浏览器渲染，他的比喻很准："你让一个画家蒙着眼睛画画，画完不让他看，然后怪他画得丑？" 他的清晨流程：醒来，手机上启动三个任务，该干嘛干嘛，晚点回来检查。规划模式起步 → 敲定计划 → 自动接受修改 → 完成多个Claude实例，一个周密计划，一个共享知识库，没了。

626

152

236K

waylongox retweeted

卡颂

@kasong2048

4 months ago

距离国内软件行业大裁员还有接近 2 年的时间为什么是接近 2 年呢？因为在AI领域，一个范式从在国外出现，到圈内热议，再到国内公司大规模应用的周期，就是 2 年。在国外 AI Coding 领域，这 3 年经历了3次范式变迁： 1. AI 辅助编程：将 AI 作为 Linter、搜索引擎、代码补全工具 2. Vibe Coding：与 AI 结对编程，也是大部分人用 Cursor 的方式 3.（当下）Harness Engineering：程序员设计强约束、可观测、可评估、可反馈、可回退的编码环境，由 Coding Agent 自主完成所有编程工作 3 的核心思想是 —— 程序员不写一行代码，如果发现 AI 写出不好的代码，第一要务不是指导他改，而是排查环境中哪一环出了问题，修复环境。举个例子：当 AI 写的前端代码有 bug，首先应该排查是不是在”可观测性“上不足（比如接入 Agent Browser 让他能自己走查页面）当前国内主流编程范式是 2，还是大量需要程序员参与。当 1 年多 2 年后，3 成为主流范式，就会出现程序员裁员潮缺粮 10% 的结果不是粮食涨价 10% ，而是一直涨价到饿死 10% 人为止。同样，主流范式进入 3 会一直让“主要工作职责能被 3 能力范围（也就是业务开发的所有编码工作）覆盖的程序员”全部失业

588

528

114K

waylongox retweeted

Huan

@Huanusa

5 months ago

42分钟，讲透一辈子的财富框架？瑞·达利欧带来了一场 42 分钟的财富创造大师课不是选股技巧不是短线暴富而是三件事：现金流结构周期判断风险分散他反复强调—— 真正的财富，不是赚得多。是穿越周期还能活着。大多数人研究“收益”。顶级投资者先研究“生存”。 42分钟听完，你会发现：赚钱只是结果，框架才是一生的差距

102

664

225K

waylongox retweeted

Huan

@Huanusa

5 months ago

李录是过去几十年里最成功的投资者之一但几乎没有公开视频直到 2006 年—— Li Lu 在 Columbia University 上了一堂估值课整整一堂课讲透“什么是价值” 没有K线没有热点只有框架与判断力这段视频几乎消失了 30 年的公开记录里却在 2006 年留下了这节“禁片级”课堂真正的投资，从来不是预测是估值值得反复观看

305

608K

waylongox retweeted

Dhairya

@dkare1009

7 months ago

Top 10 YouTube channels to learn AI & ML in 2026 1. Andrej Karpathy https://t.co/RSFn4tYrqk 2. Sebastian Raschka https://t.co/vfqkB8Jq4O 3. sentdex https://t.co/x69pFN1N7k 4. StatQuest with Josh Starmer https://t.co/ASoicNjyWO 5. Jeremy Howard https://t.co/LkT5a7C3T4 6. Krish Naik https://t.co/bWbXwJc22I 7. CampusX https://t.co/4n4mzA2TLo 8. 3Blue1Brown https://t.co/izVnMMzyKD 9. MIT OpenCourseWare https://t.co/8aWGE5Fr8v 10. Stanford Online https://t.co/RAtD6d4q1w

dkare1009's tweet photo. Top 10 YouTube channels to learn AI & ML in 2026

1. Andrej Karpathy
https://t.co/RSFn4tYrqk

2. Sebastian Raschka
https://t.co/vfqkB8Jq4O

3. sentdex
https://t.co/x69pFN1N7k

4. StatQuest with Josh Starmer
https://t.co/ASoicNjyWO

5. Jeremy Howard
https://t.co/LkT5a7C3T4

6. Krish Naik
https://t.co/bWbXwJc22I

7. CampusX
https://t.co/4n4mzA2TLo

8. 3Blue1Brown
https://t.co/izVnMMzyKD

9. MIT OpenCourseWare
https://t.co/8aWGE5Fr8v

10. Stanford Online
https://t.co/RAtD6d4q1w

277

977

44K

waylongox retweeted

柴郡🔔｜Crypto+AI Plus

@0xCheshire

7 months ago

山姆·奥特曼用短短 16 分钟，直接解释了为什么大多数初创公司会失败。他拆解了： • 产品—市场匹配 • 团队建设 • 市场选择 • 竞争护城河下面是 14 条价值超过 20 万美元 MBA 的经验教训： 1. 初创公司 80% 的成功来自一件事

328

104K

waylongox retweeted

Swapna Kumar Panda

@swapnakpanda

7 months ago

Planning to learn AI & ML in 2026? You SHOULD subscribe to these 10 YouTube channels. 1. Andrej Karpathy https://t.co/zERhSHF5jp 2. Sebastian Raschka https://t.co/ZFL0omfWuI 3. sentdex https://t.co/yuQEYdyyiV 4. StatQuest with Josh Starmer https://t.co/ss5szaEe7E 5. Jeremy Howard https://t.co/4yNWhmjZzz 6. Krish Naik https://t.co/wiGkeAkkwz 7. CampusX https://t.co/pK9hutMcnO 8. 3Blue1Brown https://t.co/2UxZFKneV1 9. MIT OpenCourseWare https://t.co/NA6WstSs3y 10. Stanford Online https://t.co/4GWPFbEGJs Add any other channels if you find them great.