Ankit Dhall

Sr. Deep Learning Engineer @NVIDIA Visual Gen AI | DL Algorithms
Prev. @latticeflowai Seervision @ETH @amzracing @motionaldrive @UniFreiburg @iiit_hyderabad VIT

[Ankit Dhall @ankitdhall]

Building generative models using libraries is great for getting started but the high-level APIs abstract away the details (usually where the devil is)

I implemented Torchsmith, an open-source package that implements generative models from scratch using basic PyTorch operations🧵

[Ankit Dhall @ankitdhall]

@ottogin1 @CVPR Thanks @ottogin1 for the effort to make it accessible to a wider audience

[Ankit Dhall @ankitdhall]

I'm excited to start a new journey at @NVIDIA as Senior Deep Learning Engineer working on Visual Generative AI in the Deep Learning Algorithms team!

[Ankit Dhall @ankitdhall]

@bose_joey Will there be a recording?

[Jia-Bin Huang @jbhuang0604]

Scrolling the AI news timeline as a researcher feels like a teenager browsing Instagram:

"Everyone else has figured everything out!"

Reliable home robots imminent, 100× productivity AI agents, insane visual generation ...

Exciting, but anxiety-inducing. What am I doing? 😬

[Ankit Dhall @ankitdhall]

Finally, a single unified text I have been waiting for.

[Yang Song @DrYangSong]

Applications change, but the principles are enduring. After a year's hard work led by @JCJesseLai, we are really excited to share this deep, systematic dive into the mathematical principles of diffusion models. This is a monograph we always wished we had.

[Ankit Dhall @ankitdhall]

@jm_alexia Was going to request for a recording but you have already taken care of it🙏

[Ankit Dhall @ankitdhall]

Looking forward to reading this one

[Alexia Jolicoeur-Martineau @jm_alexia]

New paper 📜: Tiny Recursion Model (TRM) is a recursive reasoning approach with a tiny 7M parameters neural network that obtains 45% on ARC-AGI-1 and 8% on ARC-AGI-2, beating most LLMs.

Blog: https://t.co/w5ZDsHDDPE

Code: https://t.co/7UgKuD9Yll

Paper: https://t.co/3m8ANhNMiw

[Ankit Dhall @ankitdhall]

(5/5) This thread is a concise version, check out the repo on GitHub for the full details

https://t.co/Yv5EwPpuw7

P.S.: Definitely recommend this way for anyone who wants to get a deeper understanding and see how to train and sample from such models

[Ankit Dhall @ankitdhall]

Building generative models using libraries is great for getting started but the high-level APIs abstract away the details (usually where the devil is)

I implemented Torchsmith, an open-source package that implements generative models from scratch using basic PyTorch operations🧵

[Ankit Dhall @ankitdhall]

(4/5) Flow matching:
- ODE formulation: Gaussian flow matching and Linear flow matching
- SDE formulation: Langevin flow matching

[Pedro Domingos @pmddomingos]

Matrix multiplication is now a multi-trillion dollar business.

[Jack Morris @jxmnop]

it’s been an interesting ride watching the conventional nomenclature of machine learning gradually lose all meaning. there used to be TRAIN and TEST and everything was simple. now we train on the universe. and we test on the universe, too. are we gaming our benchmarks? are we extrapolating or interpolating? if a model is trained on the entire internet but generates a single novel sentence, is it just combining phrases from Reddit or writing something truly novel? feels like we lack the words to even describe the systems we’ve built.

[Ankit Dhall @ankitdhall]

AdamW the Undisputed

[Percy Liang @percyliang]

We did a very careful study of 10 optimizers with no horse in the race. Despite all the excitement about Muon, Mars, Kron, Soap, etc., at the end of the day, if you tune the hyperparameters rigorously and scale up, the speedup over AdamW diminishes to only 10% :-( Experiments are made possible by Marin (https://t.co/UgEjGM0HPY); anyone developing new optimizers: please come try your method on this benchmark!

[Ankit Dhall @ankitdhall]

@apsarathchandar @3blue1brown +1 for recommending Gilbert Strang's lectures

[Kevin Zhang @kevinzhang25]

while doing ML research i spend a disproportionately large amount of time looking at this specific figure

[Ankit Dhall @ankitdhall]

It's ASML's world and we just live in it

[François Chollet @fchollet]

NVIDA chips are manufactured by TSMC, a Taiwanese company. They're created using EUV lithography machines manufactured by ASML, a Dutch company. These machines consist of >50% of German parts (by value), in particular ZEISS optics.

[Ankit Dhall @ankitdhall]

@giffmana @XiaohuaZhai @__kolesnikov__ @_basilM Okay, so the values were obtained via hyper parameter search.

Not clear what you mean when referring to "prior".

> "This is because the bias term ensures that the training starts close to the prior"

[Ankit Dhall @ankitdhall]

@giffmana @XiaohuaZhai @__kolesnikov__ @_basilM "We initialize t' and b to log 10 and −10 respectively. This makes sure the training starts roughly close to the prior ..."

Couldn't find more about how the two values were determined in the text.