Yi-Chen Chen

Half-time researcher, full-time human

[Andrej Karpathy @karpathy]

I packaged up the "autoresearch" project into a new self-contained minimal repo if people would like to play over the weekend. It's basically nanochat LLM training core stripped down to a single-GPU, one file version of ~630 lines of code, then:

- the human iterates on the prompt (.md)
- the AI agent iterates on the training code (.py)

The goal is to engineer your agents to make the fastest research progress indefinitely and without any of your own involvement. In the image, every dot is a complete LLM training run that lasts exactly 5 minutes. The agent works in an autonomous loop on a git feature branch and accumulates git commits to the training script as it finds better settings (of lower validation loss by the end) of the neural network architecture, the optimizer, all the hyperparameters, etc. You can imagine comparing the research progress of different prompts, different agents, etc.

https://t.co/YCvOwwjOzF
Part code, part sci-fi, and a pinch of psychosis :)

[Yi-Chen Chen @grtzsohalf]

0_0

[AK @_akhaliq]

AI is taking over

[Andrej Karpathy @karpathy]

Promising. Everyone should hope that we can throw away tokenization in LLMs. Doing so naively creates (byte-level) sequences that are too long, so the devil is in the details.

Tokenization means that LLMs are not actually fully end-to-end. There is a whole separate stage with its own training and inference, and additional libraries. It complicates the ingest of additional modalities. Tokenization also has many subtle sharp edges. Few examples:

That "trailing whitespace" error you've potentially seen in Playground? If you end your (text completion API) prompt with space you are surprisingly creating a big domain gap, a likely source of many bugs:
https://t.co/f2PBaw2iA8

Tokenization is why GPTs are bad at a number of very simple spelling / character manipulation tasks, e.g.:
https://t.co/XR3d5g4uwp

Tokenization creates attack surfaces, e.g. SolidGoldMagikarp, where some tokens are much more common during the training of tokenizer than they are during the training of the GPT, feeding unoptimized activations into processing at test time:
https://t.co/y72eaIeRrP

The list goes on, TLDR everyone should hope that tokenization could be thrown away. Maybe even more importantly, we may find general-purpose strategies for multi-scale training in the process.

[PyTorch @PyTorch]

We just introduced PyTorch 2.0 at the #PyTorchConference, introducing torch.compile!

Available in the nightlies today, stable release Early March 2023.

Read the full post: https://t.co/DFO0HojkMt

🧵below!

1/5

[Yi-Chen Chen @grtzsohalf]

Maybe one of the biggest news recently in math (?)

[Yi-Chen Chen @grtzsohalf]

https://t.co/CiIZKtAqt9

One comment below: "Man’s putting that -2022 exponent there like it’s an Olympiad problem"
loll

[Mark Tenenholtz @marktenenholtz]

Python 3.11 is out! 🎉

This is one of the most exciting releases in a while, including significant speed upgrades and better error messages.

Here's what's new:

[Yi-Chen Chen @grtzsohalf]

Collective Intelligence for Deep Learning: A Survey of Recent Developments https://t.co/53tMIMGrf2 來自 @hardmaru

[Yi-Chen Chen @grtzsohalf]

Matrix multiplication is a well-known problem with simple description and enormous influence.
AlphaTensor made an inspiring breakthrough after decades since Strassen in 1969.

[Yi-Chen Chen @grtzsohalf]

DeepMind: Discovering novel algorithms with AlphaTensor https://t.co/UkhwUpuoJF

[Jack Clark @jackclarkSF]

Talked with @TheEconomist about the industrialization of AI - a theme I've been covering for years in Import AI, and some of the implications of this which we @AnthropicAI laid out in 'Predictability and Surprise in Large Generative Models' https://t.co/6R2Qa4szOr .

[Yi-Chen Chen @grtzsohalf]

What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus https://t.co/ZRWMlWJ4xV

[hardmaru @hardmaru]

A.I. progress in a nutshell:

[Yi-Chen Chen @grtzsohalf]

This newsletter is epic "From 0 to $Bn: startup studies. How did they do it (short plans)?" Here‘s an invitation li https://t.co/z2jIBhcY9h

[Yi-Chen Chen @grtzsohalf]

@LimThingHan 我先研究一下 可以一起討論哈哈

[Yi-Chen Chen @grtzsohalf]

Requesting faucet funds into 0xaA7b1DD77CeA3c5afFA65795302ab9E06Efd40C9 on the #Rinkeby #Ethereum test network.

[Yi-Chen Chen @grtzsohalf]

@LimThingHan 研究一下而已ＸＤ

[Hung-yi Lee (李宏毅) @HungyiLee2]

Receive the YouTube Creator Silver Award for 100,000 subscribers. When I started uploading videos about DL to YouTube in the fall of 2016, I never imagined this achievement. Thanks to all subscribers. We learn together. https://t.co/R0MBtURN3f

[Akiyoshi Kitaoka @AkiyoshiKitaoka]

Vertical or horizontal borders appear to tilt.

[Daniel Probst @[email protected] @skepteis]

The circles don't move...