![PTenigma's tweet photo. HOW MUCH DATA DO YOU NEED to train an AI model to generate realistic photos of faces, or realistic videos, or medical images? How can you do it without needing trillions of images?
💡This insanely good paper is full of rich ideas for anyone training generative models https://t.co/TRPOvyfMAi
💡As I explain here ( https://t.co/95go4133KJ ) you can train a generative model using denoising diffusion models ( https://t.co/HiiGOo2vJN ) but it's not obvious how much training data you actually need - is 30,000 brains enough? What about 100,000 photographs, or will you need 100 million?
💡Without enough training data it's not going to create very realistic samples - it will memorize the training data but not jump to learn the full data distribution (often called a "phase transition" [1])
💡 The paper shows that the training set requirements are surprisingly modest for natural images and only grow linearly in the embedding dimension (see graphs)
💡 The paper starts by explaining that training a diffusion model with a denoising autoencoder (DAE) is equivalent to the "unsupervised subspace clustering problem" - so training diffusion models is essentially learning low-dimensional manifolds of the data distribution. But why do directions in this space correspond to anything meaningful (where faces can be controllably changed from male to female, or to model and alter specific attributes like hairstyle)?
💡The answer is in the Jacobian of the DAE - when we train diffusion models with the U-Net architecture on various image datasets, they find that the numerical rank of the Jacobian of the DAE, i.e., ∇_x_t x_θ(x_t , t), is substantially lower than the ambient dimension of the training data (at most time steps)
💡To practically identify these directions, they compute the Jacobian matrix of the autoencoder at a particular input image, perform an SVD (Singular Value Decomposition), and then analyze the principal components (corresponding to the largest singular values). These components can be interpreted as a low-rank controllable subspace where you can vary the images in understandable ways (see pic). These singular vectors enable different semantic edits in terms of gender, hairstyle, or color of the image (or age and disease, if you work on medical images).
💡SO..... for a simple low-rank Gaussian model, they show that training diffusion models with a DAE - to learn the data distribution - is equivalent to performing PCA on the training samples. They then show that optimizing the training loss of diffusion models is equivalent to solving the subspace clustering problem, i.e., you should maximize the sum of the squared norms of each point’s projection onto its assigned subspace.
💡Then they explain 2 reasons why you don't need insane amounts of training data to learn to generate amazingly realistic videos and images - (1) even for complex datasets like ImageNet, the intrinsic dimensionality is ~40 - much lower than its ambient dimension, and ALSO (2) the Jacobian of the DAE of diffusion models trained on real image data is low-rank for most time steps, and the parameterization captures this low-rank property. SO,,,, we can do generative AI without needing all the data on the internet (or running out of medical images). The plots show the phase transition where you have enough training data to learn the generative distribution.
🔥This paper is about as exciting as when a new paper showed that self-attention in transformers is really doing kernel PCA [2], and self-attention projects the query vectors onto the principal component axes of the key matrix.
🔥 In the same way, basically the diffusion model is doing subspace disentanglement but this time on the overall embedding.
🔥 So really a lot of this stuff seems made up ad hoc, like the crazy transformer diagram, but has a link to optimal results in low-dimensional classical linear multivariate statistics.
[1] see work by Haim Sompolinsky for more on "phase transitions" in ML and AI and how they relate to having learned something properly
[2] https://t.co/OAfKEPWOhj](https://pbs.twimg.com/media/GkblVTiWUAA3KQp.jpg)
![PTenigma's tweet photo. HOW MUCH DATA DO YOU NEED to train an AI model to generate realistic photos of faces, or realistic videos, or medical images? How can you do it without needing trillions of images?
💡This insanely good paper is full of rich ideas for anyone training generative models https://t.co/TRPOvyfMAi
💡As I explain here ( https://t.co/95go4133KJ ) you can train a generative model using denoising diffusion models ( https://t.co/HiiGOo2vJN ) but it's not obvious how much training data you actually need - is 30,000 brains enough? What about 100,000 photographs, or will you need 100 million?
💡Without enough training data it's not going to create very realistic samples - it will memorize the training data but not jump to learn the full data distribution (often called a "phase transition" [1])
💡 The paper shows that the training set requirements are surprisingly modest for natural images and only grow linearly in the embedding dimension (see graphs)
💡 The paper starts by explaining that training a diffusion model with a denoising autoencoder (DAE) is equivalent to the "unsupervised subspace clustering problem" - so training diffusion models is essentially learning low-dimensional manifolds of the data distribution. But why do directions in this space correspond to anything meaningful (where faces can be controllably changed from male to female, or to model and alter specific attributes like hairstyle)?
💡The answer is in the Jacobian of the DAE - when we train diffusion models with the U-Net architecture on various image datasets, they find that the numerical rank of the Jacobian of the DAE, i.e., ∇_x_t x_θ(x_t , t), is substantially lower than the ambient dimension of the training data (at most time steps)
💡To practically identify these directions, they compute the Jacobian matrix of the autoencoder at a particular input image, perform an SVD (Singular Value Decomposition), and then analyze the principal components (corresponding to the largest singular values). These components can be interpreted as a low-rank controllable subspace where you can vary the images in understandable ways (see pic). These singular vectors enable different semantic edits in terms of gender, hairstyle, or color of the image (or age and disease, if you work on medical images).
💡SO..... for a simple low-rank Gaussian model, they show that training diffusion models with a DAE - to learn the data distribution - is equivalent to performing PCA on the training samples. They then show that optimizing the training loss of diffusion models is equivalent to solving the subspace clustering problem, i.e., you should maximize the sum of the squared norms of each point’s projection onto its assigned subspace.
💡Then they explain 2 reasons why you don't need insane amounts of training data to learn to generate amazingly realistic videos and images - (1) even for complex datasets like ImageNet, the intrinsic dimensionality is ~40 - much lower than its ambient dimension, and ALSO (2) the Jacobian of the DAE of diffusion models trained on real image data is low-rank for most time steps, and the parameterization captures this low-rank property. SO,,,, we can do generative AI without needing all the data on the internet (or running out of medical images). The plots show the phase transition where you have enough training data to learn the generative distribution.
🔥This paper is about as exciting as when a new paper showed that self-attention in transformers is really doing kernel PCA [2], and self-attention projects the query vectors onto the principal component axes of the key matrix.
🔥 In the same way, basically the diffusion model is doing subspace disentanglement but this time on the overall embedding.
🔥 So really a lot of this stuff seems made up ad hoc, like the crazy transformer diagram, but has a link to optimal results in low-dimensional classical linear multivariate statistics.
[1] see work by Haim Sompolinsky for more on "phase transitions" in ML and AI and how they relate to having learned something properly
[2] https://t.co/OAfKEPWOhj](https://pbs.twimg.com/media/GkblUJuaoAMlJI8.jpg)
![PTenigma's tweet photo. HOW MUCH DATA DO YOU NEED to train an AI model to generate realistic photos of faces, or realistic videos, or medical images? How can you do it without needing trillions of images?
💡This insanely good paper is full of rich ideas for anyone training generative models https://t.co/TRPOvyfMAi
💡As I explain here ( https://t.co/95go4133KJ ) you can train a generative model using denoising diffusion models ( https://t.co/HiiGOo2vJN ) but it's not obvious how much training data you actually need - is 30,000 brains enough? What about 100,000 photographs, or will you need 100 million?
💡Without enough training data it's not going to create very realistic samples - it will memorize the training data but not jump to learn the full data distribution (often called a "phase transition" [1])
💡 The paper shows that the training set requirements are surprisingly modest for natural images and only grow linearly in the embedding dimension (see graphs)
💡 The paper starts by explaining that training a diffusion model with a denoising autoencoder (DAE) is equivalent to the "unsupervised subspace clustering problem" - so training diffusion models is essentially learning low-dimensional manifolds of the data distribution. But why do directions in this space correspond to anything meaningful (where faces can be controllably changed from male to female, or to model and alter specific attributes like hairstyle)?
💡The answer is in the Jacobian of the DAE - when we train diffusion models with the U-Net architecture on various image datasets, they find that the numerical rank of the Jacobian of the DAE, i.e., ∇_x_t x_θ(x_t , t), is substantially lower than the ambient dimension of the training data (at most time steps)
💡To practically identify these directions, they compute the Jacobian matrix of the autoencoder at a particular input image, perform an SVD (Singular Value Decomposition), and then analyze the principal components (corresponding to the largest singular values). These components can be interpreted as a low-rank controllable subspace where you can vary the images in understandable ways (see pic). These singular vectors enable different semantic edits in terms of gender, hairstyle, or color of the image (or age and disease, if you work on medical images).
💡SO..... for a simple low-rank Gaussian model, they show that training diffusion models with a DAE - to learn the data distribution - is equivalent to performing PCA on the training samples. They then show that optimizing the training loss of diffusion models is equivalent to solving the subspace clustering problem, i.e., you should maximize the sum of the squared norms of each point’s projection onto its assigned subspace.
💡Then they explain 2 reasons why you don't need insane amounts of training data to learn to generate amazingly realistic videos and images - (1) even for complex datasets like ImageNet, the intrinsic dimensionality is ~40 - much lower than its ambient dimension, and ALSO (2) the Jacobian of the DAE of diffusion models trained on real image data is low-rank for most time steps, and the parameterization captures this low-rank property. SO,,,, we can do generative AI without needing all the data on the internet (or running out of medical images). The plots show the phase transition where you have enough training data to learn the generative distribution.
🔥This paper is about as exciting as when a new paper showed that self-attention in transformers is really doing kernel PCA [2], and self-attention projects the query vectors onto the principal component axes of the key matrix.
🔥 In the same way, basically the diffusion model is doing subspace disentanglement but this time on the overall embedding.
🔥 So really a lot of this stuff seems made up ad hoc, like the crazy transformer diagram, but has a link to optimal results in low-dimensional classical linear multivariate statistics.
[1] see work by Haim Sompolinsky for more on "phase transitions" in ML and AI and how they relate to having learned something properly
[2] https://t.co/OAfKEPWOhj](https://pbs.twimg.com/media/GkblSyoX0AAOsy2.jpg)
Olivia
Online
Weizheng Yan
@Bruce_Leon_Plat
Lab of Neuroimaging, NIH, deep learning, neuroimaging, mental disorder.
Bethesda, MD
Joined May 2017
202 Following
74 Followers
198 Posts
Pinned Tweet
🚨New paper alert!🚨 Our latest study is now published in Journal of Neuroscience! 🧠✨ Too busy to read? No worries—We turned our key findings into a song !🎵 Give it a listen and let the science sink in: https://t.co/VzEGe6S3b2 #BrainResearch #SciComm 🎶🔬
Methylphenidate promotes a frontoparietal-dominant brain state improving cognitive performance: a randomized trial https://t.co/qSYg6jw3Se
🚨New paper alert!🚨 Our latest study is now published in Journal of Neuroscience! 🧠✨ Too busy to read? No worries—We turned our key findings into a song !🎵 Give it a listen and let the science sink in: https://t.co/VzEGe6S3b2 #BrainResearch #SciComm 🎶🔬
Who to follow
Lijun AN | 安��军
@anlijuncn
Postdoc@Jacob Vogel ⬅️ PhD@Thomas Yeo.
Neurodegenerative Disease, Multi-omics, Brain Imaging, Machine Learning
Jintao Song 宋锦涛
@jintaosong1995
son, visual working memory, visual attention, global first,
Institute of Biophysics, Chinese Academy of Sciences
Eric Wang
@ericwan53761434
Mostly posting on Bluesky now, finding me there:
https://t.co/6awBzVKFsa
Three AI-powered steps to faster, smarter peer review https://t.co/NUqnyVSRrG
A machine learning model can estimate rates of brain aging by comparing MRIs taken days to decades apart. As brain aging rate is predictive of Alzheimer’s, the technique could be helpful in assessing Alzheimer’s disease risk. In PNAS: https://t.co/7Dh70n0jYO
HOW MUCH DATA DO YOU NEED to train an AI model to generate realistic photos of faces, or realistic videos, or medical images? How can you do it without needing trillions of images?
💡This insanely good paper is full of rich ideas for anyone training generative models https://t.co/TRPOvyfMAi
💡As I explain here ( https://t.co/95go4133KJ ) you can train a generative model using denoising diffusion models ( https://t.co/HiiGOo2vJN ) but it's not obvious how much training data you actually need - is 30,000 brains enough? What about 100,000 photographs, or will you need 100 million?
💡Without enough training data it's not going to create very realistic samples - it will memorize the training data but not jump to learn the full data distribution (often called a "phase transition" [1])
💡 The paper shows that the training set requirements are surprisingly modest for natural images and only grow linearly in the embedding dimension (see graphs)
💡 The paper starts by explaining that training a diffusion model with a denoising autoencoder (DAE) is equivalent to the "unsupervised subspace clustering problem" - so training diffusion models is essentially learning low-dimensional manifolds of the data distribution. But why do directions in this space correspond to anything meaningful (where faces can be controllably changed from male to female, or to model and alter specific attributes like hairstyle)?
💡The answer is in the Jacobian of the DAE - when we train diffusion models with the U-Net architecture on various image datasets, they find that the numerical rank of the Jacobian of the DAE, i.e., ∇_x_t x_θ(x_t , t), is substantially lower than the ambient dimension of the training data (at most time steps)
💡To practically identify these directions, they compute the Jacobian matrix of the autoencoder at a particular input image, perform an SVD (Singular Value Decomposition), and then analyze the principal components (corresponding to the largest singular values). These components can be interpreted as a low-rank controllable subspace where you can vary the images in understandable ways (see pic). These singular vectors enable different semantic edits in terms of gender, hairstyle, or color of the image (or age and disease, if you work on medical images).
💡SO..... for a simple low-rank Gaussian model, they show that training diffusion models with a DAE - to learn the data distribution - is equivalent to performing PCA on the training samples. They then show that optimizing the training loss of diffusion models is equivalent to solving the subspace clustering problem, i.e., you should maximize the sum of the squared norms of each point’s projection onto its assigned subspace.
💡Then they explain 2 reasons why you don't need insane amounts of training data to learn to generate amazingly realistic videos and images - (1) even for complex datasets like ImageNet, the intrinsic dimensionality is ~40 - much lower than its ambient dimension, and ALSO (2) the Jacobian of the DAE of diffusion models trained on real image data is low-rank for most time steps, and the parameterization captures this low-rank property. SO,,,, we can do generative AI without needing all the data on the internet (or running out of medical images). The plots show the phase transition where you have enough training data to learn the generative distribution.
🔥This paper is about as exciting as when a new paper showed that self-attention in transformers is really doing kernel PCA [2], and self-attention projects the query vectors onto the principal component axes of the key matrix.
🔥 In the same way, basically the diffusion model is doing subspace disentanglement but this time on the overall embedding.
🔥 So really a lot of this stuff seems made up ad hoc, like the crazy transformer diagram, but has a link to optimal results in low-dimensional classical linear multivariate statistics.
[1] see work by Haim Sompolinsky for more on "phase transitions" in ML and AI and how they relate to having learned something properly
[2] https://t.co/OAfKEPWOhj
![PTenigma's tweet photo. HOW MUCH DATA DO YOU NEED to train an AI model to generate realistic photos of faces, or realistic videos, or medical images? How can you do it without needing trillions of images?
💡This insanely good paper is full of rich ideas for anyone training generative models https://t.co/TRPOvyfMAi
💡As I explain here ( https://t.co/95go4133KJ ) you can train a generative model using denoising diffusion models ( https://t.co/HiiGOo2vJN ) but it's not obvious how much training data you actually need - is 30,000 brains enough? What about 100,000 photographs, or will you need 100 million?
💡Without enough training data it's not going to create very realistic samples - it will memorize the training data but not jump to learn the full data distribution (often called a "phase transition" [1])
💡 The paper shows that the training set requirements are surprisingly modest for natural images and only grow linearly in the embedding dimension (see graphs)
💡 The paper starts by explaining that training a diffusion model with a denoising autoencoder (DAE) is equivalent to the "unsupervised subspace clustering problem" - so training diffusion models is essentially learning low-dimensional manifolds of the data distribution. But why do directions in this space correspond to anything meaningful (where faces can be controllably changed from male to female, or to model and alter specific attributes like hairstyle)?
💡The answer is in the Jacobian of the DAE - when we train diffusion models with the U-Net architecture on various image datasets, they find that the numerical rank of the Jacobian of the DAE, i.e., ∇_x_t x_θ(x_t , t), is substantially lower than the ambient dimension of the training data (at most time steps)
💡To practically identify these directions, they compute the Jacobian matrix of the autoencoder at a particular input image, perform an SVD (Singular Value Decomposition), and then analyze the principal components (corresponding to the largest singular values). These components can be interpreted as a low-rank controllable subspace where you can vary the images in understandable ways (see pic). These singular vectors enable different semantic edits in terms of gender, hairstyle, or color of the image (or age and disease, if you work on medical images).
💡SO..... for a simple low-rank Gaussian model, they show that training diffusion models with a DAE - to learn the data distribution - is equivalent to performing PCA on the training samples. They then show that optimizing the training loss of diffusion models is equivalent to solving the subspace clustering problem, i.e., you should maximize the sum of the squared norms of each point’s projection onto its assigned subspace.
💡Then they explain 2 reasons why you don't need insane amounts of training data to learn to generate amazingly realistic videos and images - (1) even for complex datasets like ImageNet, the intrinsic dimensionality is ~40 - much lower than its ambient dimension, and ALSO (2) the Jacobian of the DAE of diffusion models trained on real image data is low-rank for most time steps, and the parameterization captures this low-rank property. SO,,,, we can do generative AI without needing all the data on the internet (or running out of medical images). The plots show the phase transition where you have enough training data to learn the generative distribution.
🔥This paper is about as exciting as when a new paper showed that self-attention in transformers is really doing kernel PCA [2], and self-attention projects the query vectors onto the principal component axes of the key matrix.
🔥 In the same way, basically the diffusion model is doing subspace disentanglement but this time on the overall embedding.
🔥 So really a lot of this stuff seems made up ad hoc, like the crazy transformer diagram, but has a link to optimal results in low-dimensional classical linear multivariate statistics.
[1] see work by Haim Sompolinsky for more on "phase transitions" in ML and AI and how they relate to having learned something properly
[2] https://t.co/OAfKEPWOhj](https://pbs.twimg.com/media/Gkbpr9xaoAAS_Fk.jpg)
How can we use small LLMs to shift more AI workloads onto our laptops and phones?
In our paper and open-source code, we pair on-device LLMs (@ollama) with frontier LLMs in the cloud (@openai, @together), to solve token-intensive workloads on your 💻 at 17.5% of the cloud cost while maintaining 97.9% of the accuracy.
See Gru and the Minions in action below, 🔉on please (h/t @cartesia)!
📢Thrilled to introduce #MedArena https://t.co/QvH0DBdMyq 🏥, a platform for clinicians to try frontier #LLMs for free (o3, Gemini, Perplexity + more).
Compare and rank medical chatbots and help shape the future of medical AI! 🏆 Current leaderboard👇
Please share #MedTwitter
Our latest RCT on iTBS for post-stroke cognitive impairment is out in @brainstimj. High-dose iTBS (3600 pulses/day) targeting the personalized frontoparietal cognitive network is safe and more effective than lower-dose (1200 pulses/day) and sham.
https://t.co/fCteBdHWmO
Today we’re partnering with the U.S. national labs to explore how AI can accelerate scientific discovery. Over 1,000 researchers will test frontier models, including OpenAI’s o3-mini, and help shape the future of AI in science. https://t.co/MQxJ0npKjI
🚀 Call for Papers: SyntaGen @ #CVPR2025 ! 📢 Submit your work on generative models for synthetic visual datasets.
📝 Deadline: Mar 22nd! 🔗 https://t.co/qcg71KAAuu
#AI #GenerativeModels @CVPR
✨🧠Excited to have had the opportunity to present our work on Synthetic DTI Map Generation using Diffusion Models at #SPIE Conference. Some of the important points of discussion from the paper are in this 🧵
So excited to share that DeepPrep is now online in @naturemethods! 🚀 It’s 10× faster than the SOTA pipeline and more robust in handling clinical cases. We’ve just released v25.1.0 with a concise GUI and support for both Win and Linux. Give it a try! 📷 https://t.co/So57Nvu7JC
What’s the relationship between #AmyloidBeta and tau protein?
New research in @ScienceTM involving #AlzheimersDisease patient PET and fMRI data reveals amyloid-beta may drive tau spread through hyperconnectivity between vulnerable brain regions. https://t.co/604aUz6S4q
Psychedelic mental health research is booming—but how do we keep up with rapidly evolving evidence? Our team proposes a living evidence synthesis approach to ensure clinicians, policymakers, and the public have access to the latest, most reliable data.
🧵
https://t.co/rHYk6pRcxK
Today we introduce an AI co-scientist system, designed to go beyond deep research tools to aid scientists in generating novel hypotheses & research strategies. Learn more, including how to join the Trusted Tester Program, at https://t.co/1eqmTTZOLr
Many scientists who have tried OpenAI's 'deep research' tool are impressed with its ability to write literature reviews or full review papers and even identify gaps in knowledge. But others are less enthusiastic.
https://t.co/Sh73dXjKHh
Stanford’s FREE Machine Learning Courses:
CS221 - Artificial Intelligence
CS229 - Machine Learning
CS230 - Deep Learning
CS234 - Reinforcement Learning
CS224N - NLP with Deep Learning
Find course links inside:
The largest medical #AI randomized controlled trial yet performed, enrolling >100,000 women undergoing mammography screening, was published today @LancetDigitalH
The use of A.I. led to 29% higher detection of cancer, no increase of false positives, and reduced workload compared with radiologists without A.I.. https://t.co/GGrLqdG5yq
25-50% of staff at the NSF to be laid off in the next 2 months apparently. Seems bad.
Last Seen Users on Sotwe
Most Popular Users
Elon Musk 
@elonmusk
241.5M followers
Barack Obama
@barackobama
118.8M followers
Cristiano Ronaldo
@cristiano
114.5M followers
Donald J. Trump
@realdonaldtrump
111.8M followers
Narendra Modi
@narendramodi
107.1M followers
Rihanna
@rihanna
98.7M followers
NASA
@nasa
92.3M followers
Justin Bieber
@justinbieber
91.8M followers
KATY PERRY
@katyperry
90M followers
Taylor Swift
@taylorswift13
84M followers
Lady Gaga
@ladygaga
75.5M followers
Virat Kohli
@imvkohli
73.5M followers
Kim Kardashian
@kimkardashian
70.9M followers
YouTube
@youtube
68.8M followers
Neymar Jr
@neymarjr
66.5M followers
Bill Gates
@billgates
65.3M followers
Selena Gomez
@selenagomez
63.1M followers
The Ellen Show
@theellenshow
62.2M followers
CNN
@cnn
61.7M followers
X
@x
60.7M followers