🚨 EXCLUSIVE: Manchester City found guilty on virtually all charges relating to breaches of Premier League financial regulations. #MCFC expected to appeal against verdict issued by independent commission. Sanctions undecided, process ongoing @TheAthleticFC https://t.co/Muxqp4beK9
@ThomasWolgast@GurushaJuneja Hmmm I definitely didn’t take into consideration the inter-group sort of citation pushing, but you are right in saying it has become a rather ineffective incentive
We think computational depth is the missing scaling axis, i.e. we should be doing a lot more deep learning!
Every other axis has been scaled by OOMs over the past few years (params, data, sparsity, test-time reasoning), but depth has been stuck at ~100 layers since GPT-3.
We've found that LLMs are both *severely* depth-bottlenecked and bad at using the depth they have, and that architectural interventions that lift this bottleneck efficiently lead to gains that increase with compute.
w/ @akshayvegesna