Testing some CDNA code with @karparthy's llm.c on an 8xMI250X -- LGTM :) This is GPT2 with 10B tokens of fineweb-edu. Thanks to @Yuchenj_UW & @hyperbolic_labs for the machine!
I trained GPT-2 (124M) using @karpathy's llm.c for just $2.70 (~18kWh @ $0.15 per kWh) with 4x AMD 7900 XTX GPUs.
This is *5.3x* cheaper than the $14.33 it cost with 8 x H100 GPUs (run by @Yuchenj_UW).