Simo Ryu

I like cats, math and codes
[email protected]

[Simo Ryu @cloneofsimo]

This was work done last year, but my lazy ass uploaded it just this week
It was huge fun working on this project.

[Hanch Han @hanchchch]

yesterday, @cloneofsimo and I released a tech report on the video pretraining research we worked on last year. we believe sharing our experience with the community building video models will be of great help!
https://t.co/9YZmEs6lXN

[Simo Ryu @cloneofsimo]

Ok i can use my phone in the weekend
So what the fuck happened this week why is my portfolio fucked up

[Simo Ryu @cloneofsimo]

Ill be offline for three weeks, military training. Bye bye fam

[Simo Ryu @cloneofsimo]

Honestly Gary is just chad. When everyone is amazed, excited, bittersweet, awed, slightly scared of the technological progress of LLMs, he has the bulls to go "nah this aint shyt"

[Gary Marcus @GaryMarcus]

since many people are asking me about the new math results from OpenAI, here is my take.

1. It uses symbolic AI (Lean, etc) in addition to LLMs which is what i said for years we would need to do; this this confirms what I actually said (as opposed to various fictitious misrepresentations that run rampant around here )

2. It still doesn’t mean they have AGI; the math stuff AFAIK does not generalize broadly, because you can’t use the same symbolic verification (and symbolic data augmentation techniques) in the open-ended real world.

So yes I do feel vindicated wrt to neurosymbolic AI but no the G in AGI (general) is not yet solved.

What I said in 2019 remains true: neurosymbolic AI is necessary but not sufficient; world models are also key.

[Simo Ryu @cloneofsimo]

Context: LLM resolved latest theoretical bound for matmul and gradient descent

[Yuhan Ye @YyuhanYe]

The Silver Rate Is Tight for GD!

After non-trivial interaction with ChatGPT 6, we prove the lower bound n^{-(p+C\sqrt{\log\log n/\log n})}, p=\log_2(1+\sqrt{2}), for GD with predetermined nonnegative stepsizes in smooth convex optimization. The silver exponent is optimal.

[Simo Ryu @cloneofsimo]

It has to be some gods endgame plot that latest theoretical breakthrough on gradient descent and matrix multiplication came from
Gradient descent and matrix multiplication

[Simo Ryu @cloneofsimo]

LLM went from hallucinating 3-digit multiplication to fundamentally improving multiplication algorithm itself.
...In 5 years.

[Simo Ryu @cloneofsimo]

its kinda bittersweet tbh

[OpenAI @OpenAI]

We’re releasing a broad range of new mathematical results produced by an internal frontier model.

We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.

https://t.co/7N6TPlft1P

[Simo Ryu @cloneofsimo]

"Threesum is now much faster to find than we previously thought" was not the best way to explain it to my girlfriend...

[Simo Ryu @cloneofsimo]

I might be convinced that I am conscious
But how do i know if you are?
Do we just declare that as self evident or is there concrete evidence that we know for sure our fellow humans are conscious?

[Simo Ryu @cloneofsimo]

Fyi the scoring isnt elo. Elo didnt really work. Some other normalized ranking modeling also didnt work. I ended up using something like

Fij (people moving from i to j)
Tij = Fij / sumj Fij (i to j normalized per i)
Sj= sumj Tij (basically like when people leave how often do they choose j)

[Simo Ryu @cloneofsimo]

Ok i asked astra to do this. and basically
* everyone wants to move to US
* Finland is happiest but nobody wants to move there
* very slight correlation between how happy the residents are and how people want to move there.
* people want to move to russia very much (??)

[Minh Nhat Nguyen 🦭 @menhguin]

i think a lot of "which places are the best to live" should just be migration/emigration ELO. it's the ultimate ranking of revealed preference. the same applies to workplaces.

[Simo Ryu @cloneofsimo]

Btw number papers with topic of ai safety in neurips 2025, is 2.2%. So under this estimation slutcon has 5x more overlap to ai safety community than the AI community itself. AI Safety is extremely important and becoming more and more so, but this association has got to disappear.

Like im fairly certain its some group of people that associate themselves with ai safety that are not even professional researchers and permanently online larpers on lesswrong that are causing this.

[Aella @Aella_Girl]

Unfortunately the overlap you perceive on here I suspect is mostly incorrect. People are actively lying about slutcon. My best guess is 10% of the attendees were associated with x risk in some way? I'm sorry that people are causing a false perception for others.

That being said, slutcon super emphasizes consent, we have consent practices and teach a bunch of boundary respecting and handling rejection, and imo if more AI ppl went to slutcon I expect this would overall improve the quality of women's experiences with them.

[Simo Ryu @cloneofsimo]

@noahsolomon Adorable!

[Simo Ryu @cloneofsimo]

Pubg is my third favorite game and I am consistently surprised they have such a big AI effort

[Kangwook Lee @Kangwook_Lee]

🧵 It's finally here! The full technical report on PUBG Ally is finally on Arxiv: https://t.co/C1ULXBYVSQ

Sharing a few highlights in the thread!

[Simo Ryu @cloneofsimo]

@yunus_eee So true

[Simo Ryu @cloneofsimo]

??????????? ~40% improvement on whats already sub 1min record?

[Larry Dial @classiclarryd]

New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s!

This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it.
Specifically:
-(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time.
-Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used.
-Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2.
-Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step.
-Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params.
-Hand-rolled flash attention for 64 dim heads.

There are several additions that add accuracy too:
-(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15.
-(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application.
-A couple additional dynamic skip connections in the network.

The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large,
only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead.
https://t.co/Ycrzy6JFC3

As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: https://t.co/YEfM1VpOTi

[Simo Ryu @cloneofsimo]

I sometimes cant believe 18k people will have to watch Lavender eat his favorite thing, unwillingly.

[Simo Ryu @cloneofsimo]

Will probly intended as sarcasm but it might just be true....

[will brown @willcb]

@nicksaraev I am not joking when I say this.

Tinder, Hinge, and dating in general is over.

Porn is now very, very good. You just don't know yet because all porn circulating today is from ~2-3 generations ago.

PornHub et al has solved infinite personalized gooning.

[Simo Ryu @cloneofsimo]

Crazy to think 99% of the download is probably chatgpt / claude

[Three.js @threejs]

19M npm downloads / week 🚀