My intuition is that an AI with anything close to today’s capabilities in mathematical problem solving could not have been produced by training only on pre-1859 math
Not just trying to be cute here; I think it’s genuinely hard to separate the capabilities from the “knowledge.” Even if we could get a current model to forget 20th century math, I think the answer depends on how much you ablate the instincts (i.e. about what kinds of tactics are likely to be fruitful) that it learned from training on that math
I don’t think this is quite the right way to think about it
Even setting aside Levent’s contributions, the AI proof makes use of tons of concepts developed by humans, much of it specifically for this type of problem
Clearly humanity’s collective effort loosened the jar
The answer actually sheds some light on why SBF thinks you should be willing to flip a slightly biased coin for a chance to double the world
He says it's equivalent to flipping a coin to prevent one of two worlds from being destroyed, in which case you should shut up and multiply (you have a 50% chance of surviving if you don't flip the coin, and 51% chance of surviving if you do)
IMO this depends on some very unusual beliefs about population ethics, including that not bringing someone into existence is as bad as killing someone, that—paradoxically and ironically—people should not be altruistic with respect to other worlds (he says "to each world, it doesn't matter what happens to the other world"), and that no form of @willmacaskill's "saturation view" is correct (https://t.co/uTJyapt2y2)
The fact of the interview is interesting but the questions are well-chosen and heavily-researched, recommended
In particular I'm glad someone followed up on SBF's degenerate beliefs about coin flips
EXCLUSIVE: I'm 14, and I just interviewed Sam Bankman-Fried.
We talked about:
– Whether FTX was worth it
– His p(doom)
– Why he only put $500M into Anthropic
– Whether he’d flip a 51/49 coin for his own freedom
– How FTX almost became a political prediction market
– Why he thinks we should “get rid of the feds”
– What he’d do if he were my age
The Supercycle: The Sam Bankman-Fried Interview is out now.
00:00 Highlights and introduction
01:52 Life in prison and the justice system
11:09 Anthropic and the future of AI
20:00 Investing in Anthropic and running FTX Ventures
29:25 Effective altruism and the FTX Future Fund
35:33 AI safety and the race to superintelligence
40:32 AI timelines, extinction risk, and mind uploading
50:43 Prediction markets and the future of finance
57:22 Books, learning, and entertainment
1:00:22 The coin flip that could destroy the world
1:06:52 Slowing AI and the limits of expected value
1:14:38 What actually helps AI safety?
1:17:40 Was FTX worth it?
1:19:28 Advice for young people in the AI era
We will inevitably cross a threshold where depending entirely on humans to write and review a complex proof becomes more unreliable than depending entirely on AI
Indeed I think we’ve already crossed it
We audited Masullo’s manuscript The Music of the Primes, which claims an unconditional proof of the Generalized Riemann Hypothesis.
In its current form, the paper's argument has fundamental problems in several distinct parts of the strategy.
🧵
1/
I wonder if the Association for Human Mathematics will just refuse to ever accept a proof of RH now, and "human mathematics" becomes an independent formal system with its own assumptions
People write papers that declare a theorem "provable in HM"
Btw Alessandro Masullo proposed a proof of GRH several months ago that has been under review by Brian Conrad "... and it is so far looking good." https://t.co/mHup5jGfd5
after posting the Publication Boom report, I started chatting with the arXiv staff and ended up helping them with a bunch of data work
I'm releasing these internal reports now for any uber academics curious about the arXiv publishing pipeline
some highlights:
are you reading this? researchers would give up ~half their team to be able to use models from the past year! (5.3 was released in february). in just the past year, the models have become worth that much.
this is one of the best uplift stats we've seen. the compute bullets are notable as well, as they give a sense of how automated R&D and compute compare as inputs to algorithmic progress.
credit to everyone who pushed for sharing this
Because alignment is a really difficult and tempting research problem, I think too many “Manhattan project for safe AI” concepts focus on it
ASI is extremely dangerous and disruptive even if it is fully aligned