before blaming the architecture, overfit a tiny batch... if the model cannot nearly memorize a handful of examples, there's probably something wrong with the data, objective, masking, optimization, or implementation
@zdogmode I feel that's the AI bottleneck.. since AI needs good labelled data to train itself and we're generating so much AI slop and contaminating the internet that it's hard to get raw data and train a better version of AI