Jonny Schult

Software Engineer, philosophy teacher, Peircian Pragmatist and cat enthusiast.

[Jonny Schult @JonnySchult]

Be a patriot. Read C. S. Peirce, 🇺🇲's best philosopher. "Few persons care to study logic, because everybody conceives himself to be proficient enough in the art of reasoning already.
[Reasoning] isn't so much a natural gift as a long and difficult art." https://t.co/Vh4kJqX0p9

[Jonny Schult @JonnySchult]

Millennia of philosophy gave us traction on intelligence. Decades of scientific research gave us an engineering program. The months and years ahead will flood society with AI applications. Ethics and policy have little time to catch up. 🤖
https://t.co/QoNMb8IrFZ

[Jonny Schult @JonnySchult]

Arooj Aftab on Colbert:
https://t.co/XkFgFOqVLs

[Jonny Schult @JonnySchult]

https://t.co/vkXLq3PXJw via @qobuz
There's stunningly beautiful, thoughtful, and human music from artists and genres outside of the mainstream (there's great music in the mainstream, too). If you have some time, Arooj Aftab's album is exceptionally deserving of a listen.

[Dario Amodei @DarioAmodei]

Machines of Loving Grace: my essay on how AI could transform the world for the better

https://t.co/CFGd4Sbq86

[Jonny Schult @JonnySchult]

@RadarHits 5r4t

[Jonny Schult @JonnySchult]

While I haven’t given much thought to analysis or critique, the claims seem baldly plausible. Some version of this future seems sufficiently likely that all ought to immediately ponder its significance.

[Jonny Schult @JonnySchult]

15. This may all be set against a backdrop of WWIII, but the winner of SI will be the winner overall. If society makes it through this, then it is incumbent on us to share the rewards of SI with assurances of no future SI development by other nations.

[Jonny Schult @JonnySchult]

That’s it. Many of the assumptions are naturalist, based on realpolitik, and empirically informed. Moreover, Leopold says this isn’t most deeply a set of normative claims, but descriptive claims (gird thineself).

[Jonny Schult @JonnySchult]

14. The Project is a Manhattan Project-style government takeover of AI development with security set to the max. That is air-gapped data centers with developer vetting and limited, surveyed leave, etc. Weed out spies, combine labs, bunker-ize, and containerize the supercluster.

[Jonny Schult @JonnySchult]

13. Since these aren’t acceptable, it is of paramount importance that we maintain our substantial lead. Perhaps the immediate steps are regulation and intervention to require high security by AI companies, but the long term (in the next couple of years) is to start The Project.

[Jonny Schult @JonnySchult]

12. We have two bad outcomes: domination by intolerable, CCP values, or trying to develop SI without safeguards and potentially turning on Skynet without knowing that it is safe. The former is likely cultural genocide, and the latter plausibly existentially terminating.

[Jonny Schult @JonnySchult]

11. When this happens, the competitive nature of the race will leave alignment measures in a prisoner’s dilemma scenario. We can’t trust that the CCP or other parties won’t be reckless in their development practices, and we can’t tolerate their victory, so we have to be reckless.

[Jonny Schult @JonnySchult]

10. But the biggest threat is the CCP. We’re likely already handing them the all-important neural network weights of our present and upcoming generations of AI. If the CCP realizes the full import of this, the race is on.

[Jonny Schult @JonnySchult]

I wrapped up June by reading @leopoldasch's https://t.co/RvayjwcZyH. If you haven't already, you should do it this July.
My interpretive understanding of its basic claims and assumptions on the initial read is in the thread.

[Jonny Schult @JonnySchult]

9. Good news! We’re way ahead technologically and in terms of talent. Bad news! The AI development labs' security is abysmal. Even Google’s somewhat security-minded development admits its security is porous enough that a rogue state like North Korea or terrorists could hack it.

[Jonny Schult @JonnySchult]

8. Ensuring this slow pace of development is difficult when we’re in a close winner-takes-all race with other parties seeking the advantages of SI. The mother nation of SI will have a decisive military and geopolitical advantage. Worry #2: Authoritarians and rogues desire this.

[Jonny Schult @JonnySchult]

7. We have to get this right. It takes time to do that, and we should iterate slowly on rolling out new generations of AI until confidence is high that alignment has been achieved. Facing an SI testing scenario with ambiguous results, time is needed to implement safeguards.

[Jonny Schult @JonnySchult]

6. Our laissez-faire development strategies are insufficient. Landing on the right SI implementation by chance is unlikely. If we create a misaligned SI, scenarios range from SI's benign neglect (how we treat non-tasty animals) on the happy path to SI being a nightmare factory.

[Jonny Schult @JonnySchult]

5. The risks, however, are many. The essay explores two in broad strokes. Worry #1. Instrumentalist reasoning is no guarantor of good values or morality. The SI cluster-being we’re Frankensteining isn’t tied to humanity’s values out of necessity. Its values have to be engineered.

[Jonny Schult @JonnySchult]

4. This will take massive resources, but the forces are already marshaling the necessities for 100b+ and even trillion+ dollar superclusters. The promise of SI is significant along every axis of economic and political import, potentially 10xing economies and military might.

[Jonny Schult @JonnySchult]

3. How quickly? AGI is as close to us in the future as Squid Games is in the past, and with the feedback cycles of AGI agents being employed to create the next generation of AI, we could see SI in this decade. The 2020s, that is.

[Jonny Schult @JonnySchult]

2. As such, there is no special gate that prevents AI from becoming superintelligent. The trendlines show that artificial general intelligence (AGI) is right around the corner. AGI will then iterate to superintelligent AI (SI) very quickly.