Varun Nagaraja

Just curious

[Varun Nagaraja @varun_nagaraja]

Implementing numerically stable softmax is a classic AI coding interview question. Next time, one-up your interviewer with Jon's version.

[Jon Barron @jon_barron]

Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking: