baochens

[Google DeepMind @GoogleDeepMind]

Huge congratulations to @DemisHassabis and John Jumper on being awarded the 2024 Nobel Prize in Chemistry for protein structure prediction with #AlphaFold, along with David Baker for computational protein design.

This is a monumental achievement for AI, for computational biology, and science itself. 🧬

[Zeyuan Allen-Zhu, Sc.D. @ZeyuanAllenZhu]

If you're attending ICML 2024, join my 2-hour tutorial on Monday July 22 to explore the Physics of Language Model - all 6 parts. Visit: https://t.co/Nz7v73gTqR and it will be live-streamed on Zoom. BONUS: this is the premiere of Part 2.1 + 2.2, don't miss out! #ICML2024 #MetaAI

[Jon Barron @jon_barron]

The legendary Ross Girshick just posted his CVPR workshop slides about the 1.5 decades he spent ~solving object detection as it relates to the ongoing LLM singularity. Excellent read, highly recommended. https://t.co/i3qS4iheU5

[Durk Kingma @dpkingma]

The recording of our talk for the ICLR'24 test-of-time award (with @wellingmax) is now available online:
https://t.co/uY5tG2aCUd

Biggest live audience I've ever spoken to, with >2000 attendees 😅. But it was a lot of fun!

[Jeff Dean @JeffDean]

Gemma 2 is now available available in 9B and 27B sizes (great for practical deployments) 🎉

These open-source models offer best-in-class performance (often better than significantly larger models), run at incredible speed across different hardware and easily integrate with other AI tools.

Download the weights on @Kaggle and @HuggingFace, or access the models in Google AI Studio

Both sizes also do very well compared to other proprietary and open models on lmsys (although error bars are still largish due to limited votes so far):
See more on the blog:
https://t.co/0WrtElgh8e

Tech report: https://t.co/RCREbHRIse

Congrats to all the people who have been working hard on this new release!

[Saining Xie @sainingxie]

Introducing Cambrian-1, a fully open project from our group at NYU. The world doesn't need another MLLM to rival GPT-4V. Cambrian is unique as a vision-centric exploration & here's why I think it's time to shift focus from scaling LLMs to enhancing visual representations.🧵[1/n]

[Jeff Dean @JeffDean]

Nice thread on how to use the long context capabilities of Gemini 1.5 Pro to understand complex code bases. Various approaches like:

"draw me an overall diagram of the major modules of the system"

"draw me neural net architectures for any sub models used"

"give me a list of files and summaries of what they do"

"list external components used"

"describe future roadmap of possible enhancements"

"what should I read to better understand the ideas behind the code"

"what are other open source repositories similar to this one"

etc.

[Andrej Karpathy @karpathy]

📽️ New 4 hour (lol) video lecture on YouTube:
"Let’s reproduce GPT-2 (124M)"
https://t.co/QTUdu8b0qh

The video ended up so long because it is... comprehensive: we start with empty file and end up with a GPT-2 (124M) model:
- first we build the GPT-2 network
- then we optimize it to train very fast
- then we set up the training run optimization and hyperparameters by referencing GPT-2 and GPT-3 papers
- then we bring up model evaluation, and
- then cross our fingers and go to sleep.
In the morning we look through the results and enjoy amusing model generations. Our "overnight" run even gets very close to the GPT-3 (124M) model. This video builds on the Zero To Hero series and at times references previous videos. You could also see this video as building my nanoGPT repo, which by the end is about 90% similar.

Github. The associated GitHub repo contains the full commit history so you can step through all of the code changes in the video, step by step.
https://t.co/BOzkxQ8at2

Chapters.
On a high level Section 1 is building up the network, a lot of this might be review. Section 2 is making the training fast. Section 3 is setting up the run. Section 4 is the results. In more detail:
00:00:00 intro: Let’s reproduce GPT-2 (124M)
00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint
00:13:47 SECTION 1: implementing the GPT-2 nn.Module
00:28:08 loading the huggingface/GPT-2 parameters
00:31:00 implementing the forward pass to get logits
00:33:31 sampling init, prefix tokens, tokenization
00:37:02 sampling loop
00:41:47 sample, auto-detect the device
00:45:50 let’s train: data batches (B,T) → logits (B,T,C)
00:52:53 cross entropy loss
00:56:42 optimization loop: overfit a single batch
01:02:00 data loader lite
01:06:14 parameter sharing wte and lm_head
01:13:47 model initialization: std 0.02, residual init
01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms
01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms
01:39:38 float16, gradient scalers, bfloat16, 300ms
01:48:15 torch.compile, Python overhead, kernel fusion, 130ms
02:00:18 flash attention, 96ms
02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms
02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping
02:21:06 learning rate scheduler: warmup + cosine decay
02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms
02:34:09 gradient accumulation
02:46:52 distributed data parallel (DDP)
03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)
03:23:10 validation data split, validation loss, sampling revive
03:28:23 evaluation: HellaSwag, starting the run
03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro
03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA
03:59:39 summary, phew, build-nanogpt github repo

[Thomas Wolf @Thom_Wolf]

This might be one of the most important 45-mn read you could indulge in today if you want to understand the secret behind high performance large language models like Llama3, GPT-4 or Mixtral

Inspired by the @distillpub interactive graphics papers, we settled to write the most extensive, enjoyable and in-depth tech report we could draft on the science of creating high quality web-scale datasets, detailing all the steps and learnings that came in our recent 15 trillion tokens 🍷FineWeb release

And it's not all, in this article we also introduce 📚FineWeb-Edu a filtered subset of Common Crawl with 1.3T tokens containing only web pages with very high educational content. Up to our knowledge, FineWeb-Edu out-performs all openly release web-scale datasets by a significant margin on knowledge- and reasoning-intensive benchmarks like MMLU, ARC, and OpenBookQA

To close, we make a number of surprising observations on the "quality" of the internet it-self which may challenge some of the general assumptions on web data (not saying more, I'll let you draw your conclusions ;)

Take a walk through it, you won't be disappointed 👇

[Yi Ma @YiMaTweets]

Increasingly, I start to believe there are four ordered categories of computation: what is Computable, what is Tractable, what is Scalable, and what is Natural. The last category consists of operations that can truly be utilized by Nature with limited space, time, and resource.

[Yi Ma @YiMaTweets]

Our 100-page mega paper “White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?” https://t.co/oNzWzumjRf was accepted by the Journal of Machine Learning Research. How fast things have progressed since its initial posting, almost made results in it outdated… That’s what happens once you have a pertinent theory.

[Jeff Dean @JeffDean]

One other thing in the updated Gemini 1.5 Pro report: we show how a research model that is a mathematics-specialized version of 1.5 Pro achieves a record score of 91.1% on the MATH benchmark (the SOTA just 3 years ago, in May, 2021 was 6.9%!).

[Lei Zhang @leizhangcs]

Today, we're thrilled to announce the release of Grounding DINO 1.5, featuring two distinct models tailored for various scenarios:

🌟 Grounding DINO 1.5 Pro offers top-tier zero-shot performance on diverse benchmarks: 54.3 AP on COCO, 55.7 AP on LVIS, and 30.2 AP on ODinW.

[Google DeepMind @GoogleDeepMind]

We’re introducing new additions to Gemma: our family of open models built with the same technology as Gemini.

🔘 PaliGemma: a powerful open vision-language model
🔘 Gemma 2: coming soon in various sizes, including 27 billion parameters

→ https://t.co/KYNJ1O4xfL #GoogleIO

[Demis Hassabis @demishassabis]

We think of @GoogleDeepMind as the engine room of @Google in the AI era. Thrilled to share our vision at #GoogleIO incl the latest Gemini model 1.5 Flash, Project Astra our universal AI agent effort, our new gen video model Veo, Imagen 3 & lots more! https://t.co/lcsgr7ptJK

[Jim Fan @DrJimFan]

I know your timeline is flooded now with word salads of "insane, HER, 10 features you missed, we're so back". Sit down. Chill. <gasp> Take a deep breath like Mark does in the demo </gasp>. Let's think step by step:

- Technique-wise, OpenAI has figured out a way to map audio to audio directly as first-class modality, and stream videos to a transformer in real-time. These require some new research on tokenization and architecture, but overall it's a data and system optimization problem (as most things are).

High-quality data can come from at least 2 sources:

1) Naturally occurring dialogues on YouTube, podcasts, TV series, movies, etc. Whisper can be trained to identify speaker turns in a dialogue or separate overlapping speeches for automated annotation.

2) Synthetic data. Run the slow 3-stage pipeline using the most powerful models: speech1->text1 (ASR), text1->text2 (LLM), text2->speech2 (TTS). The middle LLM can decide when to stop and also simulate how to resume from interruption. It could output additional "thought traces" that are not verbalized to help generate better reply.

Then GPT-4o distills directly from speech1->speech2, with optional auxiliary loss functions based on the 3-stage data. After distillation, these behaviors are now baked into the model without emitting intermediate texts.

On the system side: the latency would not meet real-time threshold if every video frame is decompressed into an RGB image. OpenAI has likely developed their own neural-first, streaming video codec to transmit the motion deltas as tokens. The communication protocol and NN inference must be co-optimized.

For example, there could be a small and energy-efficient NN running on the edge device that decides to transmit more tokens if the video is interesting, and fewer otherwise.

- I didn't expect GPT-4o to be closer to GPT-5, the rumored "Arrakis" model that takes multimodal in and out. In fact, it's likely an early checkpoint of GPT-5 that hasn't finished training yet.

The branding betrays a certain insecurity. Ahead of Google I/O, OpenAI would rather beat our mental projection of GPT-4.5 than disappoint by missing the sky-high expectation for GPT-5. A smart move to buy more time.

- Notably, the assistant is much more lively and even a bit flirty. GPT-4o is trying (perhaps a bit too hard) to sound like HER. OpenAI is eating Character AI's lunch, with almost 100% overlap in form factor and huge distribution channels. It's a pivot towards more emotional AI with strong personality, which OpenAI seemed to actively suppress in the past.

- Whoever wins Apple first wins big time. I see 3 levels of integration with iOS:
1) Ditch Siri. OpenAI distills a smaller-tier, purely on-device GPT-4o for iOS, with optional paid upgrade to use the cloud.
2) Native features to stream the camera or screen into the model. Chip-level support for neural audio/video codec.
3) Integrate with iOS system-level action API and smart home APIs. No one uses Siri Shortcuts, but it's time to resurrect. This could become the AI agent product with a billion users from the get-go. The FSD for smartphones with a Tesla-scale data flywheel.

[Andrej Karpathy @karpathy]

Nice new read on tokenization!
You've heard about the SolidGoldMagikarp token, which breaks GPT-2 because it was present in the training set of the Tokenizer, but not the LLM later.

This paper digs in in a lot more depth and detail, on a lot more models, discovering a less extreme version of the above - partially-trained tokens in both open/closed models. You have to be careful with a lot of small details and implications - weight sharing, constants in residual streams, weight-decays, regex splitting patterns, BPE, UTF-8, etc.

TLDR Tokenization remains a major pain and a large LLM attack surface. Including these partially-trained tokens in your prompts drifts the model out of distribution into undefined regions of the dynamics, areas that the model is not used to. They confuse the LLM. The paper's focus is discovery and not engineering, but it seems likely one can find "token attacks" that reliably induce target weirdness: pop-off safety, alter personality or behaviors (?), any other kind of ... otherwise undefined behavior, whatever that may look like.

Now go ask GPT-4 about _ForCanBeConverted, $PostalCodesNL, useRalative, and _typingsJapgolly :)
(or see Figure 4 of the paper at the very end for simple examples)

[Philipp Schmid @_philschmid]

The Deepseek v2 paper holds some gems.💎 Last Week, @deepseek_ai v2 was released a 236B parameter MoE with a 128k context window and 21B active parameter, and now the paper is available.👀

Architecture:
Decoder with Multi-head Latent Attention (MLA) and a Mixture-of-Experts (MoE). MLA reduces key-value cache demands during inference by compressing keys and values into a latent vector.

Training:
📚 Pretrained on 8.1T tokens, mostly English and Chinese, with a sequence length of 4096.
🎓 Supervised fine tuning (SFT) on 1.5M samples, 1.2M for helpfulness, and 0.3M for safety.
🏆 Used Group Relative Policy Optimization (GRPO) to align the model outputs with human preferences, especially focusing on instruction following.

Learning Strategy (warmup-and-step-decay strategy):
⬆️ Linear Learning rate from 0 to the maximum value (2.4e-4) during the first 2K steps (warmup period).
⬇️ After training about 60% of tokens, the learning rate is reduced and multiplied by 0.316.
⬇️ After training about 90% of tokens, the learning rate is again reduced by 0.316.

Other insights:
📈 Used batch size scheduling from 2304 to 9216 for the first 225B tokens.
🌐 Used YaRN to extend the context window from 4K to 128K.
💰 42.5% reduced training cost compared to Deepseek 67B due to sparse activation
🏆 MMLU: 78.5 ; AlpacaEval 2.0: 38.9; MT-Bench: 8.97 🔧 Used Pipeline Parallelism, Expert Parallelism, and Data Parallelism for distributed training.
🎯 GRPO used a multi-reward framework with rewards from helpful, safety, and rule-based rewards.
🚀 Used hybrid engine with vLLM inference backend for RLHF training

[Christian Szegedy @ChrSzegedy]

@ducha_aiki @ylecun @JeffDean @ahmadyan @deliprao @karpathy @AndrewYNg @woj_zaremba I just like simple things: using x-entropy for discrete decisions and L2 for continuous regression. I have never really needed any other loss and both of them have clear mathematical motivation.

I think there is a certain value to being consistent and simple as much as possible.

[Christian Szegedy @ChrSzegedy]

Thanks a lot Ian, greatly appreciated!

This is the first result of my 12 years of ML carrier and the one I am still most proud of, on the other hand I found this paper did not do justice to it.

The "lessons learned" slide at the end of the presentation might be worth checking out, especially.

https://t.co/SaKIOLRuVZ