Manuel

[Paul Graham @paulg]

Whoah, self-driving cars compete with airlines. I never considered that till now.

[davidad 🎇 @davidad]

It seems odd that there’s a rough societal consensus that 1+x=0 needs to have a solution—and that it’s not just an imaginary number to appease the accountants—but 1+x²=0 need not have a solution, unless it’s an imaginary number to appease the physicists and electrical engineers.

[FFmpeg @FFmpeg]

@aakashxsh They did

[Arthur Mensch @arthurmensch]

Looking forward to building frontier open source AI models together with @Nvidia as we join the Nemotron Coalition and start training the first base models.

[John Carmack @ID_AA_Carmack]

I know there is some overlap between open source and anti-AI activists, but I have a hard time reconciling it. My million+ open source LOC were always intended as a gift to the world. Yes, I would make arguments about how it would strengthen our communities, and the GPL would prevent outright exploitation by our competitors, but those were to allay fears of my partners to allow me to make the gift.

AI training on the code magnifies the value of the gift. I am enthusiastic about it!

Some people do look at open source as a tool for social change, career advancement, or reputation building, but those are all downstream of the gift.

[Michael Nielsen @michael_nielsen]

A footnote from a current draft

[Andrej Karpathy @karpathy]

LLM OS. Bear with me I'm still cooking.

Specs:
- LLM: OpenAI GPT-4 Turbo 256 core (batch size) processor @ 20Hz (tok/s)
- RAM: 128Ktok
- Filesystem: Ada002

[Philip Rosedale @philiprosedale]

This is so beautiful. And so important to study. This phenomena (emergent coherence) is central to much of life, and most people don't even imagine it could happen.

[Andrej Karpathy @karpathy]

I packaged up the "autoresearch" project into a new self-contained minimal repo if people would like to play over the weekend. It's basically nanochat LLM training core stripped down to a single-GPU, one file version of ~630 lines of code, then:

- the human iterates on the prompt (.md)
- the AI agent iterates on the training code (.py)

The goal is to engineer your agents to make the fastest research progress indefinitely and without any of your own involvement. In the image, every dot is a complete LLM training run that lasts exactly 5 minutes. The agent works in an autonomous loop on a git feature branch and accumulates git commits to the training script as it finds better settings (of lower validation loss by the end) of the neural network architecture, the optimizer, all the hyperparameters, etc. You can imagine comparing the research progress of different prompts, different agents, etc.

https://t.co/YCvOwwjOzF
Part code, part sci-fi, and a pinch of psychosis :)

[Mitchell Hashimoto @mitchellh]

We need corporate PR speak like this to stop. Execs need to talk to people like people. The dev community is filling in the blanks anyways, whether they're right or wrong. Just own your direction and move forward, don't hide behind corporate cowardice. https://t.co/hFQtJD7ZHH

[François Chollet @fchollet]

The point of science is to cover the greatest number of empirical facts at the least model complexity cost.

[Tim Dettmers @Tim_Dettmers]

It seems the closed-source vs open-weights landscape has been leveled. GPT-5 is just 10% better at coding than an open-weight model you can run on a consumer desktop and soon laptop. If Anthropic cannot come up with a good model, then we will probably not see AGI for a while.

[Jack Morris @jxmnop]

this paper blew my mind when i read it four years ago. it blew my mind again when i reread it this morning

basic finding is that transformers pretrained on TEXT are also useful for IMAGES (and proteins, and numerical operations).... 🤯

i think the term "Universal Computation Engines" is actually a really useful mental model

my hypothesis is that transformers learn a lot of generally-useful internal mechanisms like how to load information from weights and aggregate information across positions. these tools are useful regardless of modality

in the future, the mechinterp work (SAEs, Circuits) might be able to explain this sort of thing. but for now it's just a mystery.

and in general it speaks to the eerie staying power of the Transformer. if somebody ever found a better arch, it would have to be even *more* Universal. good luck with that

[Aaron Gokaslan @SkyLi0n]

Just opened a PR yesterday that will reduce the binary size PyTorch by 40% by adding 1 flag to NVCC

With ~50M monthly of downloads of Pytorch, this one change will reduce global internet traffic by ~20PB. High impact changes like this is why I love OSS.

https://t.co/IQcAlwOuNF

[Jonathan Gorard @getjonwithit]

All parallel operations can be performed sequentially, but not all sequential operations can be performed in parallel. This failure to parallelize is what we call "causality".

(Holds true for general relativity, quantum information, Petri nets, monoidal category theory, etc.)

[Mitchell Hashimoto @mitchellh]

None cared, honestly. The topic of OSS never came up one way or the other. I think you can see this in the post-BSL and fork world too, the big cloud providers still mostly back the original afaik. They don't care. They just want the consumption. Any act that they care is performative to get more users. They don't care.

[Lisan al Gaib @scaling01]

A few more observations after replicating the Tower of Hanoi game with their exact prompts:

- You need AT LEAST 2^N - 1 moves and the output format requires 10 tokens per move + some constant stuff.
- Furthermore the output limit for Sonnet 3.7 is 128k, DeepSeek R1 64K, and o3-mini 100k tokens. This includes the reasoning tokens they use before outputting their final answer!

- all models will have 0 accuracy with more than 13 disks simply because they can not output that much!

- the max solvable sizes WITHOUT ANY ROOM FOR REASONING (floor(log2(output_limit/10)))
DeepSeek: 12 disks
Sonnet 3.7 and o3-mini: 13 disks

- If you actually look at the output of the models you will see that they don't even reason about the problem if it gets too large:
"Due to the large number of moves, I'll explain the solution approach rather than listing all 32,767 moves individually"

- At least for Sonnet it doesn't try to reason through the problem once it's above ~7 disks. It will state what the problem and the algorithm to solve it and then output its solution without even thinking about individual steps.

- it's also interesting to look at the models as having a X% chance of picking the correct token at each move
- even with a 99.99% probability the models will eventually make an error simply because of the exponentially growing problem size

[François Chollet @fchollet]

Long-distance running is one of the defining abilities of our species (which we developed for endurance hunting). It has shaped much of our biology, e.g. the loss of our fur (to facilitate thermoregulation), our uniquely long legs, high lung capacity, etc.

Go run for a few hours

[kepano @kepano]

1940s: the detonation of nuclear bombs contaminates nearly all steel on Earth, resulting in the need to scavenge low-radiation pre-war steel from sunken shipwrecks

2020s: ChatGPT contaminates content with synthetic text, resulting in the need to segregate pre-LLM sources

[Manuel @makuche_]

@giffmana @BorisMPower https://t.co/6EnVdD9zR8