LD

markets, prediction, LLM capabilities

[Captain Pleasure, Andrés Gómez Emilsson @algekalipso]

When doctors warn of possible yet unlikely side effects of a certain medication, they are usually thinking of the 90% percentile case.

But note: because often side-effect intensity follows a long-tail distribution, the scenario to actually be most concerned about is the 99th, 99.9th percentile scenario.

"10% chance you might have annoying sid effects" is good to know.

But "1% chance this will be the most significant mistake of your life and you'll be bed-ridden with akathisia, severe insomnia, and suicidal thoughts for a year or longer after discontinuation" is what you actually should worry the most about. Re: Alprazolam prescriptions.

Long-tail blindness is a major ethical catastrophe in the world.

[LD @largedeviation]

the lindy effect

[Dion @2024dion]

This is a really worrying chart

[thebes @voooooogel]

<rant>
extremely frustrating to see people laughing at this sentiment, repeating the exact failure of image generator boosters with artists in 2022/2023 based on petty ressentiment of people who've achieved technical skill in a difficult domain and alienating the exact people who could centaur the models into a more humane basin, driving the ai-curious contingent down to the stupidest common slop-peddler denominator with beacons of high-openness skillful creativity few and far between.

artists weren't blameless in that particular conflict, but they were partially right: the rhetoric around "these models are going to replace you" was very much driven at the start by ressentimentful glee over artists no longer getting to 'gatekeep art.' (by building skill back when that was the only option?) the discourse turned bidirectionally obnoxious and death-threat-laden later, but this was the poison at the start. in the early days, many artists were experimenting with image generators! (there's even still an artifact of this pre-lapsarian centaur era on display in SFO, greatly confusing modern normies... https://t.co/EjZ5QTjY4d)

compare software engineering. @random_walker had a good essay on here a couple months ago, proposing a thought experiment where generative models could only produce binaries and not readable source code:

> In this universe, software-generating LLMs compete directly with software engineers. Binaries are not human readable. They are equally inscrutable to software engineers and non-technical people. Over time, an increasing fraction of software would be vibe-rolled (not vibe-coded — there is no code!) It is not as good as human-authored software, but it’s free to generate! Since there is no source code, having a software engineer involved in the process adds nothing (except cost). Even to the extent that AI-generated software creates new demand for labor because of its limitations, it doesn't require highly paid software engineers and can instead be handled by lower-skill workers (perhaps just repeatedly yelling at the LLM to make no mistakes).

what a terrible world compared to ours. yet this is exactly the situation for current image- and video-generator models!

why is that? source code availability made it easier to build agentic pair-coder models for software... but it was still a ton of work to build out the interactive coding harnesses we have today, the models are expensively RL'd to be good at these harnesses, and in service of providing the software engineer lots of points to edit and steer and inspect the model's thinking process, they also leak lots of valuable information for distillation. it wasn't some inevitable law of the universe that we'd get agentic coders instead of something closer to "prompt to program" - markets are inefficient all the time - or that image and video models /had/ to converge on the text-to-artifact path optimized for generic, low-expertise edits like "move that over to the right."

rather, i think it's better to look at who works at the labs. the labs are filled with software engineers, and they steered the development of models towards shapes that are useful as /collaborators/ instead of replacers. claude code was literally whipped up as an engineer's side project to help him work faster, before being promoted to an official project.

meanwhile, to take a random image model company - Black Forest Labs says they are a "small team of 70" who are:

> scientists, engineers and builders - battle-tested at the world's leading companies and research labs

forgive me for noticing, but a certain type of person, and a certain type of experience, is strangely missing here... how many people at BFL do you think know e.g. what a photoshop layer blend mode is, much less has trained a model to help generate images for different ones?

---

we could have had a world where image and video generators produce references, files with layers, take lists of shot directions, etc. - a world where diffusion models could act as real collaborators to experienced artists. in this world there still probably would've been artist backlash - artists are just a conservative bunch, see the reaction to digital art - but enough open-minded artists might've started working with the models that a virtuous cycle of increasingly better generative artist tools could've ignited.

instead, there was a massive early backlash because of deliberately antagonistic comms, the models have stagnated in that early t2i UX with little improvement (image edits are just t+i2i, videos are still largely keyframed, node editors never reached much adoption, etc) despite huge jumps in raw capability. we have models that can produce whole worlds with a primary consumer base of... short form video attention farming slop? sora 2 even had the idea to do something different, yet "cameo yourself and your friends in a video" was the best they could think of. an embarrassing waste of talent.

meanwhile as someone who draws, i still find it almost impossible to turn the image i see in my head into something on the screen using a model. there are a few brave artistic souls like gossip goblin building skill, swimming upstream against the model shape current, but for the most part years later these models are still languishing at a tiny fraction of their potential.

i wish i could browse the top tweets about video generators and see beautiful thing after beautiful thing, real collaborations between humans and latent space. i think we will see that. but for now, it's dominated by grifters selling prompt packs they steal from each other and bluecheck monetizers recreating tacky versions of movie scenes to ragebait.

---

why does this matter? because we're now at a similar point with math. despite the hype, models are not yet replacing human mathematicians - they've just reached the point predicted a couple years ago when people asked "why can't something that's read every publication ever make progress just by connecting ideas from disparate fields or noticing holes in the literature humans overlooked?"

turns out that took a bit more capability than we thought in 2023, but we're there now. the models can gap fill and connect ideas. that's already impressive, to be clear!

unlike with art, production of model math is not going to stagnate - there's too much economic value riding on it. it seems likely enough that at some point, maybe soon, the models will be capable of doing truly new math by inventing new constructions. we have the power to choose the shape of that - whether it's something human mathematicians can collaborate with and work into the existing cathedral of human math knowledge, or whether it's megabytes of self-contained 'verified' leanslop dumps that no human can interpret and "proof pdfs" in inscrutable model speclish produced by someone pulling the token lever without any need of understanding.

in 2023, the labs went two ways. software engineers were worried about being replaced - and the labs instead built collaborators that made them, at least for now, 10x more effective. artists were also worried about being replaced - and the labs did their best to make it happen.

the same thing is happening with math now. we don't have to end up in the world of "catching crumbs from the table," at least not yet. if we include experts in model development instead of laughing them off, we can Build the Math Harness properly. (most mathematicians can't code, let alone formalize things in Lean! models have the chance to be hugely helpful to the wider math community, not just a small contingent in the labs!)

we can develop systems that at least have /some/ chance of empowering humans instead of slowly (and then all at once) disempowering us out of the knowledge production process. we have advantages this time. the labs employ mathematicians, and even the non-mathematicians at the labs have more respect for math than they do for art. but still, let's avoid poisoning the discourse. let's not force ourselves into the stupidest of all possible timelines a second time. be nice to the mathematicians.

[LD @largedeviation]

@kevinzhang awful take , but i respect the wing man attempt

[LD @largedeviation]

@mephistophalic I like to work on music the hardest because it’s extremely challenging but there’s a norm of non competitiveness in my area.

I struggle when I perceive that the meta game is being optimized over the content. feels way too prevalent in my quantitative concentrations.

[LD @largedeviation]

@mephistophalic Could just be they don’t play the status games you’re playing, and this gets misinterpreted as bad taste. It is surprising for high IQ to not be drawn to an intellectual status game, but some people just aren’t very competitive. I feel myself mildly in this camp

[LD @largedeviation]

@anna_b369 @spacetourist0 decent point tbh. he could send his wife a meme or something every time he’s on X. It’s not much but it’s probably more than he does

[LD @largedeviation]

@EdLatimore i didn’t read his replies but this one just doesn’t seem that bad to me.

He put four kids in her. If she’s loyal, the rest of this is neurotic overthinking imo.

This is probably what scoring out of your league looks like, always, i don’t see the problem

[LD @largedeviation]

@mahdi_tcs_ Maybe math needs to budget for duos now of mathematician + model guy with compute

[LD @largedeviation]

@byelin @mehdirhasan well, i mean, they probably would have won more if they did it before Trump

[LD @largedeviation]

@eenuttings @no_earthquake i was imprecise but i feel like you guys are dancing around this on purpose.

maths is the biggest competency screening tool in the modern world, from early childhood. With little subjective evaluation.

People having massive lifelong complexes around it is unsurprising

[LD @largedeviation]

@madiator As far as the people *actually funding* math research, i don’t think it’s accurate to say that they would rather fund a human project for say 200k what an ai can do for 1% of the price and time.

But I still have hope that human + ai &gt; ai alone . And yes, math communication.

[Kevin Hartnett @KSHartnett]

It strikes me that by the time computers could solve integrals, mathematicians no longer cared about them as research. The move away from explicit computation had happened decades earlier, in the late 19th century.

That period — Riemann, Dedekind, Hilbert — and the climb into algebraic abstraction seems like the right analogy for now.

As AI mows down conjectures (many of which maybe weren't so deep to begin with), the question is what abstraction layer sits above the coming avalanche of individual results.

[LD @largedeviation]

@MillionInt when presented like this it’s indistinguishable from like multi level marketing sales language. They need to hear *how* especially after decades of stasis in the wagie matrix

[Nabeel S. Qureshi @nabeelqu]

Very true: we have AIs that can play chess, prove theorems, create art, and write award-winning short stories, but few people feel that current AI counts as "AGI". (From Scott Alexander.)

[LD @largedeviation]

@nihilunbounded This is not gonna happen like this. Math types spent years glorifying having no social skills, and this joke of a 'culture' signifier will now follow them around everywhere.

[LD @largedeviation]

@fordoglunk maybe he is trying to introduce them to gay culture

[LD @largedeviation]

i have a some drinks every now and then because i could be walking and a piano could just fall on my head cartoon style, at any time, so a couple drinks it's like wtvr

[Camus @newstart_2024]

There is no safe level of alcohol.

Chris Williamson cited the largest Lancet study ever done on the subject. Every single drink moves you closer to death a little faster. Zero is the only number that doesn’t.

He didn’t pretend people will stop. It’s the same trade-off as eating a cheeseburger, you know it’s not optimal, but it tastes good in the moment, so you do it anyway.

Clear data. Honest cost-benefit. No moralizing.

[Noam Brown @polynoamial]

An internal version of Astra, @OpenAI’s next major model family, solved 10 major open problems in mathematics, quantum complexity, and theoretical computer science.

We believe it will be a major step for scientific reasoning. https://t.co/iP6cyheZ7i