Benet Oriol Sàbat

PhD Student @UCLA working on explicit 3D control for image/video generation.

Intern @AmazonScience

[Benet Oriol Sàbat @osbenet]

@bilawalsidhu referring to computer graphics as inverse vision from now on

[Benet Oriol Sàbat @osbenet]

@KBlueleaf @__tinygrad__ hmm how feasible is FSDP in this case

[Benet Oriol Sàbat @osbenet]

@KBlueleaf @__tinygrad__ awesome thanks

[Benet Oriol Sàbat @osbenet]

@KBlueleaf @__tinygrad__ how much faster? any significant improvement in latency? other than throughput?

[Benet Oriol Sàbat @osbenet]

@cloneofsimo did your cat take into account that even if the DFT weights increase with number of pixels, the mean square error also should divide by the number of pixels, since it's a spatial mean? perhaps preventing mse from blowing up?

[Benet Oriol Sàbat @osbenet]

@ostrisai interesting! what is the convergence speedup vs say Adam? also I wonder about the connection with running average since negative + positive numbers will average to 0

[Benet Oriol Sàbat @osbenet]

@KBlueleaf Makes sense, do you use native Torch pipe for that or any other library / custom implementation? Thanks for the advice!

[Benet Oriol Sàbat @osbenet]

@KBlueleaf I see - do you have a preferred way to parallelize training across GPU over PCIe? Other than DDP?

[Benet Oriol Sàbat @osbenet]

@KBlueleaf I mean bottleneck your training performance

[Benet Oriol Sàbat @osbenet]

@KBlueleaf in parallel settings like FSDP, doesn't the lower ddr memory generation in epyc 7003 series & slower cpu memory controller bottlneck your communication between GPUs?

[Benet Oriol Sàbat @osbenet]

@ostrisai makes sense, I wonder how that affects the training dynamics of the unconditional diffusion (when conditions are dropped)

[Benet Oriol Sàbat @osbenet]

@cloneofsimo I think you are right, theoretically. You can't ignore the fact that CoT definitely works really well, but I think the reason it works might currently be not so well understood and over-anthropomorphised. it might just come down to using more flops or more functional complexity

[Benet Oriol Sàbat @osbenet]

@Mascobot @a16z Have you tried enabling peer to peer PCIe communication between 4090s?
My understanding is sometimes the bottleneck is not in the actual PCIe bus but the memory controller of the CPU, so going through RAM is the bottleneck and you could get full PCIe utilisation by using p2p.

[Benet Oriol Sàbat @osbenet]

@ostrisai shouldn't the result of unshuffle-x16 be (16384, 512)? Perhaps you mean factor of 32?

[Benet Oriol Sàbat @osbenet]

@__tinygrad__ which Geona model? and did you enable p2p communication between 5090s?

[Benet Oriol Sàbat @osbenet]

@ostrisai Makes sense, thanks for sharing. btw, what is your go-to dataset for training these controlNets?

[Benet Oriol Sàbat @osbenet]

@jon_barron all images and videos are 3D data, since are projections of 3D a world. if we can insert the right inductive biases into image models maybe we implicitly learn good 3D foundation models with no 3D annotation.
arguably current models are already doing so to some extent.

[Benet Oriol Sàbat @osbenet]

@natanielruizg @bilawalsidhu https://t.co/UNZjIxd9DU

[abdou @thatabdou]

after running the road runner scenario through a monocular depth estimation model (that probably has much less accuracy than the real FSD system would have).