🐐 Qi Wang(Levi)

Neuroscience PhD.Student @MPICybernetics @uni_tue #MRI #ML #UnsupervisedLearning #GenerativeModels

[🐐 Qi Wang(Levi) @LeviWQ1]

@jm_alexia @goodfellow_ian @bouzoukipunks I really enjoyed your ragan paper, similar stories here but still seeking chance with perseverance

[Mathelirium @mathelirium]

I was watching a lecture on YouTube by David Tse from Stanford (see link below), and he quoted his advisor Bob Gallager: "Good theory should prune rather than grow the knowledge tree."

To demonstrate what Gallager meant, we takes one idea… "how do you move mass from one shape to another as cheaply as possible?" and watch 200 years of mathematics cut that idea down to its clean geometric core.

Monge (1781, "Mémoire sur la théorie des déblais et des remblais")
A mound on the left, a trench on the right, and every grain of earth has to choose one destination and stick to it. In the animation that’s strict one-to-one matching: each point marches to a single partner, no splitting, no sharing. It’s beautiful but rigid and hard to work with.

Kantorovich (1942, "On the Translocation of Masses"; 1948, "On a Problem of Monge")
Kantorovich relaxes the rules. Instead of forcing each point to pick exactly one target, he allows mass to split: part of it can go here, part of it can go there. On screen you see packets of mass spraying from one source cell to several targets along smooth arcs. The problem becomes a clean convex optimisation problem.

Brenier (1991, "Polar factorization and monotone rearrangement of vector-valued functions"); McCann (1995, "Existence and uniqueness of monotone measure-preserving maps"; 1997, "A convexity principle for interacting gases")
Bernier then shows that, in the most natural cost setting, the best way to move mass always comes from a hidden height function whose slopes tell you where to send things. McCann shows that many natural energy functionals behave nicely along the paths generated this way. In the animation you see contour lines of that hidden landscape, with particles gliding along the most economical path between the two shapes.

Jordan-Kinderlehrer-Otto (1998, "The variational formulation of the Fokker-Planck equation")
JKO change the question from "what is the best single shuffle?" to "how does a whole cloud evolve in time?". They show that a familiar diffusion-with-drift equation can be reinterpreted as steepest descent of a free energy in the space of probability distributions. In the scene, a blob both smooths out and gets pulled toward the embankment while a free-energy counter steadily drops.

Ambrosio-Gagli-Savaré (2005, "Gradient flows in metric spaces and in the spaces of probability measures")
Ambrosio-Gigli-Savaré take that idea and generalise it far beyond this one setting. They build a theory of gradient flows on abstract metric spaces: you only need a notion of distance and an energy, and you can talk about curves of maximal slope. Wasserstein spaces become one important example among many. In the final scene the probability field evolves in the top panel while its energy traces a clean descending curve below.

[Mathelirium @mathelirium]

In 1996, James Sethian showed something almost unfair...you can find shortest routes through a messy world by letting a wave expand once...no trial paths, no search beams, just one growing front.

Here’s how: we solve for an arrival-time field T(x,y) so that T literally means how long the wave needs to reach this point. The rule is ||∇ T|| = 1/F, where the medium is fast (F large) the front sprints, where it’s slow it trudges, and obstacles are speed ≈ 0, so the front wraps around them because that’s the only way forward.

Then comes the satisfying part: once T exists, a path doesn’t need to search at all...drop a bead anywhere and let it follow ẋ ∝ -∇ T, it slides downhill on the time landscape and traces a globally fastest route back to the source. This “wave = optimal control” viewpoint is exactly what Tsitsiklis (1995) made precise from the Hamilton-Jacobi side...compute the value/arrival-time function and the optimal trajectories fall out from it.

#FastMarching #EikonalEquation

[🐐 Qi Wang(Levi) @LeviWQ1]

@LodestoneRock Are you using annealed Gaussian noise as regularization?

[🐐 Qi Wang(Levi) @LeviWQ1]

@mariyaivasileva Back to days, they called AI “cybernetics”.

[🐐 Qi Wang(Levi) @LeviWQ1]

@mariyaivasileva Missing the old time, fixing errors the whole day to leash caffe training using FCN…

[Lucas Beyer (bl16) @giffmana]

Nice survey of papers working towards NNs with somewhat practical realistic Lipschitz bounds:

[Dayal Kalra @dayal_kalra]

Excited to share our paper "Universal Sharpness Dynamics..." is accepted to #ICLR2025!

Neural net training exhibits rich curvature (sharpness) dynamics (sharpness reduction, progressive sharpening, Edge of Stability)- but why?🤔

We show that a minimal model captures it all!
1/n

[AK @_akhaliq]

Block Diffusion

Interpolating Between Autoregressive and Diffusion Language Models

[Gabriel Peyré @gabrielpeyre]

Min/max games are convex/concave saddle points. Can be solved by primal-dual algorithms. Converges to Dirac mass strategies only when K has a single saddle point, otherwise selects mixed strategies. https://t.co/BXczoUKf7X

[Valeriy M., PhD, MBA, CQF @predict_addict]

Feeling math expert? Try to crack these ones.

#math

[Jeffrey Scholz @Jeyffre]

I read Google's paper about their quantum computer so you don't have to.

They claim to have ran a quantum computation in 5 minutes that would take a normal computer 10^25 years.

But what was that computation? Does it live up to the hype?

I will break it down.🧵

[Gabriel Peyré @gabrielpeyre]

Optimal transport, convolution, and averaging define interpolations between probability distributions. One can find vector fields advecting particles that match these interpolations. They are the Benamou-Brenier, flow-matching, and Dacorogna-Moser fields. [NB: for Dacorogna-Moser the field is singular bc of the abrupt change of support in the cat-donut example]

[Gabriel Peyré @gabrielpeyre]

I have updated my course notes on Optimal Transport with a new Chapter 9 on Wasserstein flows. It includes 3 illustrative applications: training a 2-layer MLP, deep transformers, and flow-matching generative models. You can access it here: https://t.co/m5nVCIpZYG

[Kirill Neklyudov @k_neklyudov]

so happy to see that Action Matching finds its applications in physics, outperforming diffusion models and Flow Matching!

wonderful work by Jules Berman, Tobias Blickhan, and Benjamin Peherstorfer!
https://t.co/d8KFbFRgEU

[Gabriel Peyré @gabrielpeyre]

Slides for a general introduction to mathematical tools in AI. https://t.co/09jrip0OE1

[Gabriel Peyré @gabrielpeyre]

Oldies but goldies: Edward N. Lorenz, Deterministic nonperiodic flow, 1963. Lorenz attractor is the set of limit trajectories for a simplified mathematical model for atmospheric convection. It has a fractal geometry. https://t.co/TVkuMpbEep

[Gabriel Peyré @gabrielpeyre]

There exists a single increasing map between two distributions, it is the optimal transport, and it is defined from the cumulative functions. https://t.co/PAukdgSF69

[Saining Xie @sainingxie]

Representation matters.
Representation matters.
Representation matters, even for generative models.

We might've been training our diffusion models the wrong way this whole time. Meet REPA: Training Diffusion Transformers is easier than you think! https://t.co/lyWwiTYjEt(🧵1/n)

[Gabriel Peyré @gabrielpeyre]

Saddle points come with lots of flavors. Quadratic saddle are non-degenerate and repulse gradient descent. Higher-order saddle points can attract gradient descent in a lot of directions. https://t.co/kKEtkY5bFc https://t.co/V2QD5GMhTv