Yen Ting Lin

Staff scientist at the Information Sciences Group (CCS-3), Los Alamos National Laboratory

[Yen Ting Lin @dblueeye]

@andrew_n_carr Hi, you might find our recent work interesting! https://t.co/Ufy6Ys4WVX

[Accepted papers at TMLR @TmlrPub]

Are We Really Learning the Score Function? Reinterpreting Diffusion Models Through Wasserstein Gr...

An Vuong, Michael Thompson McCann, Javier E. Santos, Yen Ting Lin.

Action editor: Jes Frellsen.

https://t.co/WmrQ1UFyLB

#diffusion #flow #generative

[Yi Ma @YiMaTweets]

9.11 is still larger than 9.8, despite can memorize solutions to PhD level questions. Again, memorizing is not understanding and knowledge is not intelligence.

[François Chollet @fchollet]

The question of whether LLMs can reason is, in many ways, the wrong question. The more interesting question is whether they are limited to memorization / interpolative retrieval, or whether they can adapt to novelty beyond what they know. (They can't, at least until you start doing active inference, or using them in a search loop, etc.)

There are two distinct things you can call "reasoning", and no benchmark aside from ARC-AGI makes any attempt to distinguish between the two.

First, there is memorizing & retrieving program templates to tackle known tasks, such as "solve ax+b=c" -- you probably memorized the "algorithm" for finding x when you were in school. LLMs *can* do this! In fact, this is *most* of what they do. However, they are notoriously bad at it, because their memorized programs are vector functions fitted to training data, that generalize via interpolation. This is a very suboptimal approach for representing any kind of discrete symbolic program. This is why LLMs on their own still struggle with digit addition, for instance -- they need to be trained on millions of examples of digit addition, but they only achieve ~70% accuracy on new numbers.

This way of doing "reasoning" is not fundamentally different from purely memorizing the answers to a set of questions (e.g. 3x+5=2, 2x+3=6, etc.) -- it's just a higher order version of the same. It's still memorization and retrieval -- applied to templates rather than pointwise answers.

The other way you can define reasoning is as the ability to *synthesize* new programs (from existing parts) in order to solve tasks you've never seen before. Like, solving ax+b=c without having ever learned to do it, while only knowing about addition, subtraction, multiplication and division. That's how you can adapt to novelty. LLMs *cannot* do this, at least not on their own. They can however be incorporated into a program search process capable of this kind of reasoning.

This second definition is by far the more valuable form of reasoning. This is the difference between the smart kids in the back of the class that aren't paying attention but ace tests by improvisation, and the studious kids that spend their time doing homework and get medium-good grades, but are actually complete idiots that can't deviate one bit from what they've memorized. Which one would you hire?

LLMs cannot do this because they are very much limited to retrieval of memorized programs. They're static program stores. However, can display some amount of adaptability, because not only are the stored programs capable of generalization via interpolation, the *program store itself* is interpolative: you can interpolate between programs, or otherwise "move around" in continuous program space. But this only yields local generalization, not any real ability to make sense of new situations.

This is why LLMs need to be trained on enormous amounts of data: the only way to make them somewhat useful is to expose them to a *dense sampling* of absolutely everything there is to know and everything there is to do. Humans don't work like this -- even the really dumb ones are still vastly more intelligent than LLMs, despite having far less knowledge.

[Yen Ting Lin @dblueeye]

Full poster can be found at https://t.co/99zEk3zzdi

[Yen Ting Lin @dblueeye]

Just made a virtual poster presentation of our #ICML24 paper "Liouville Flow Importance Sampler"

@YIFENGTIAN1

https://t.co/fdwTHzW52b

[Yann LeCun @ylecun]

Yes, I've made this point many times.
The beginning of a sigmoid looks like an exponential.

Not only can we "never be fully certain that what we are observing isn't in fact following a logistic trend before the inflection point", we can always be fully certain that *every* *single* *exponential* *trend* eventually passes an inflection point and saturates into a sigmoid.
Continuing an exponential trend beyond that inflection point requires a paradigm shift.

No physical process can grow indefinitely.
There are always friction terms in the dynamics equation that eventually become dominant (energy consumption, heat dissipation, quantum effects, thermal fluctuations, communication bandwidth, mass/energy density....).

Even processes that *appear* exponential on a long time scale are actually a succession of sigmoids, in which each new sigmoid is caused by a paradigm shift.
A good example is Moore's Law. It is saturating right now. But the exponential progress of the last 7 decades is due to a succession of technological paradigm shifts that weren't pre-ordained.
Each paradigm behaved like a sigmoid. Each new sigmoid overtook the previous one. The envelope turned out to be exponential.
We haven't seen similar paradigm shifts in, say, airplane speed or space travel.
Technological paradigm shifts require scientific breakthroughs.

[Yen Ting Lin @dblueeye]

Liouville Flow Importance Sampler (LFIS), an innovative flow-based sampler that achieved state-of-the-art performance over a range of test problems. https://t.co/YBR7hLvB0D
@YIFENGTIAN1
#ICML2024

[Yen Ting Lin @dblueeye]

Eyes on her feet... the so-called "AI".

[Mukul Sharma @stufflistings]

Meet OpenAI Sora, an AI model that can create realistic and imaginative scenes from text instructions.
#OpenAI #ChatGPT #Sora #AI

[Yen Ting Lin @dblueeye]

Our Blackout paper (ICML 23) made LANL front page!
@zachrfox @javier_e_santos @LosAlamosNatLab
https://t.co/BMHKBPhF5K

[Melanie Mitchell @MelMitchell1]

Who else is countering the LLM "AGI" narrative? Google product manager for Bard!

(From https://t.co/5QjyBtr59j by @daveyalba)

👀 ⬇️

[Luca Ambrogioni @LucaAmb]

Our paper "Spontaneous symmetry breaking in generative diffusion models" was accepted at @NeurIPSConf 2023!

We found that the generative capabilities of diffusion models are the result of a phase transition!

Preprint: https://t.co/JDVx3y7EMd
Code: https://t.co/Vx4k5zojKG

[Yen Ting Lin @dblueeye]

Yesterday, I gave a technical presentation to @LucaAmb’s group on Blackout Diffusion (https://t.co/ASutxHqzQE). The group was so kind to record the talk and share with netizens ⬇️ Special thanks to @gisilvs for organizing the presentation!

[Gianluigi Silvestri @gisilvs]

Fantastic presentation from @dblueeye on diffusion models with discrete diffusion process 👏👏 very clear explanation and promising results for their 'Blackout Diffusion'. Take a look at the presentation here https://t.co/4dTC04fOAM

[Levi @levikul09]

R² is a widely used measure of fit, but for many analysts, it is just a number.

They believe high R² ➡ Good predictions.

This is not always true!

Now I will clarify. 🔽

R-squared measures how well the regression model fits the observed data.

To be more precise: It is the proportion of the variation in the dependent variable that is predictable from the independent variable.

It usually ranges from 0 to 1: (In rare cases it can be negative, I will explain this in another tweet)

R² = 0

The model does not explain any of the variability in the dependent variable ➡ No predictive power ➡ Bad model.

R² = 1

The model perfectly explains all the variability in the dependent variable ➡ Perfect fit to the data ➡ Good model if not overfitted and has predictive power.

A high R-squared value does not mean that the predictions made by the model will be correct.

It doesn't measure predictability power, it measures how well the model fits!

In the example below, we compare the mean of the data to a fitted line.

Of course, the mean of values is not a good fit ➡ the errors are large.

On the other hand, the fitted line has smaller errors ➡ The R² will be close to 1.

To calculate R² we need:

- The total sum of squares for the mean
- Sum of squares for the residuals from the model
- Finally, subtract the ratio from 1

___

That's it for today.

I hope you've found this Tweet helpful.

Like/Retweet for support and follow @levikul09 for more Data Science content.

Thanks 😉

[Jasmine Hughes @Jas_Hughes]

The fact that there are multiple candidates for what this linear regression is blows my mind 🤯

(Original, of course: https://t.co/N87MPaNh3g)

[Mohammed AlQuraishi @MoAlQuraishi]

We built a new diffusion protein design model named Genie. We preprinted it a while ago (soon after RFDiffusion and Chroma preprints) but kept mum due to embargo. Final ICML version (major update) with code https://t.co/0wLXWMeuJU and paper here https://t.co/j977j9ddna (1/7)

[Yen Ting Lin @dblueeye]

@C_Weniger Let us try the small networks in chemistry first 😉 Dark matter sounds interesting, we’re definitely interested to know more!

[Yen Ting Lin @dblueeye]

@C_Weniger Great minds think alike! The whole purpose of developing this theory was for graph generation. We are allocating funding for it. BTW we are open to collaboration!

[Yen Ting Lin @dblueeye]

Finally, we developed a mathematical theory for arbitrary Markov jump processes on discrete support, analogous to Anderson (82) for Itô/Diffusion processes, addressing (Q3).

Thanks for your interest!

[Yen Ting Lin @dblueeye]

Blackout Diffusion: Generative Diffusion Models in Discrete-State Spaces

https://t.co/a7e81f5Qd6

Accepted by #ICML2023 @javier_e_santos @zachrfox @LosAlamosNatLab

🧵

[Yen Ting Lin @dblueeye]

As such, we concluded that the forward=noisifying and reverse=denoising (Q2) is an inadequate concept to understand GDMs. More aggressively put, GDMs should not be referred to as Denoising Diffusion Models.