There's a broadly held misconception in AI that methods that scale well are simple methods -- even, that simple methods usually scale. This is completely wrong.
Pretty much none of the truly simple methods in ML scale well. SVM, kNN, random forests are some of the simplest methods out there, and they don't scale at all. Meanwhile "train a transformer via backprop and gradient descent" is a very high-entropy method, easily 10x more complex than random forest fitting. But it scales very well.
Further, given a simple method that doesn't scale, it is usually the case that you alter it to make it scale by adding a lot of complication. For instance, take a simple a simple combinatorial search-based method (not scalable at all) -- you can make it scale by adding deep learning guidance (which blows up complexity). Scalability usually belongs to high-entropy, complex systems.
>Be Bryan Johnson
>Decide at 42 that death is optional if you have enough money and no personality
>Hire a team of 30 doctors to monitor your every function
>Eat 1,977 calories of carefully engineered beige paste per day
>Take 111 supplements
>Go to bed at 8:30pm
>Inject yourself with your own son's plasma like some kind of polite vampire
>Spend $2 million per year on this
>Biological age reduced by approximately 5 years after 3 years of effort
>Announce you are "reversing aging"
>Look somehow older than when you started
>Go on podcasts to explain that you no longer experience joy, but your telomeres are excellent
>Announces optimal body temperature is now 35°C
>Says it’s “metabolically efficient”
>Essentially refrigerated