@paraschopra 3/3
In hindsight, it seems like Sohl might've overfitted to his proxy and missed applying Occam's razor or some regularization. Given this is published in 2022, i think this probably played some role.
@paraschopra 2/3
In 2020, Ho et al. improved on Sohl's work with DDPM by relaxing some of the constraints. That made training much more stable and led to the amazing results we see now with diffusion models.
@paraschopra 1/3
Interesting! Sohl-Dickstein's 2015 work on diffusion models laid the foundation for a lot of what we see today. But his final formulation of variational inference approach struggled with stability in training.
@NirantK@OpenAIDevs Sadly these probabilities are not really calibrated well 🫤. Usually you will end up with 100% probability for classification tasks.
@o_v_shake I have seen that there's a lot of diversity in the ml-system design interviews.
Some resources that i feel are good to brush through.
- https://t.co/7pJir4R39I
- Exponent ml-sys design playlist
- Chip Huyen
@o_v_shake At a micro-level that seems to be case but I am not sure about the macro trends. Furthermore, I think the top tech's still usually prefer MTech's or a good pedigree.