A significant portion of current AI "discourse" is less about technological capabilities and more about frontier lab employees navigating their own self-esteem and sense of identity.
One of the most controversial takes I have is that ppl do not fundamentally want a world without hierarchy - they simply want a hierarchy whose scoring function rewards the traits they possess & recodes their rivals’ advantages as moral defects.
If you destroy aristocracy, ppl will rank themselves by wealth. Get rid of wealth & they rank by education levels, taste, political purity, beauty, suffering, proximity to power, technical competence, or even conspicuous indifference to status all together.
Status determines who receives / “deserves” attention, forgiveness, romantic access, institutional credibility, & the right to write history. Ofc material resources matter, but humans will often sacrifice material welfare to avoid symbolic subordination.
This begs the question: whose traits will the next social order classify as admirable?
There’s the permanent overclass / underclass post-ASI world answer - that those who own / control / remain economically complementary to AI versus everyone else - but this is itself somewhat biased by limited perspective on how the world will receive ASI (and also an artifact of projecting today’s status logic forward).
ASI may rewrite the scoring function entirely. Intelligence becomes cheap and technical competence obsolete and wealth less meaningful in at least one potential outcome. Traits like beauty / social coordination / authenticity / biological and genetic rarity could become the new basis of rank. What’s uncertain and the determining factor here is which human qualities remain scarce enough to warrant reverence in an age of abundance.
Someone who says merit is above all else usually has a particular theory of merit in mind. Likewise, someone who says lived experience matters most is also proposing an epistemic hierarchy. Rebels think courage is best, bureaucrats procedural fluency, technologists intelligence, aristocrats lineage, and revolutionaries idealogical virtue. Each of the aforementioned imagines that their preferred ranking principle is not merely advantageous but JUST.
None of this is to say injustice is fake or imaginary or equality before the law is pointless or every moral conviction is secretly fraudulent. It simply means the aspiration to eliminate hierarchy is an incoherent delusion. The humane objective post ASI will be to prevent any one individual hierarchy from becoming total.
A free society should contain many partially independent status systems. Losing one game because you’re not maxxed out on the be all end all trait shouldn’t mean losing everything in life simultaneously.
One of AI’s deepest dangers imo is therefore scalarization.
Models will become extraordinarily good at inferring latent traits from user behavior / human interaction. The danger here is if employers / platforms / lenders / governments / social networks all begin consulting variations of the same hidden representation. Then 100s of local imperfect escapable reputations collapse into one interoperable estimate of human worth / value.
It’d result in something resembling totalitarianism but without an official dictator or doctrine, just a universally queried embedding.
The system would know that you are low-ranked before anyone could explain why. Every institution would independently reach the same conclusion because they were not, in fact, independent. Your past would generalize perfectly and permanently.
So I think pluralism is kind of a protection against a single ranking function acquiring enough resolution to govern the whole, but then again, that might be obvious.
A lot of people (myself included) went into ML research because they thought weren't smart enough for pure math or theoretical physics. Now, AI is slowly but clearly starting to conquer those fields.
Goes to show that the most productive thing you can do is often to leverage yourself with tools. Humans invented spears and worked in groups to hunt the much more powerful Mammoths. We can use elementary math to invent algorithms and let GPUs go brr to solve the hardest and deepest problems in math and physics.
@Mononofu@JensenHuang This is a false equivalence. Everyone has the right to keep their code private. The problem is when someone tries to stop OTHERS from open sourcing.
@Mononofu@JensenHuang This is a false equivalence. Everyone has the right to keep their code private. The problem is when someone tries to stop OTHERS from open sourcing.