General Intelligence

Two-sided synthetic data marketplace for AI model training on @SuiNetwork & @solana. Built by @dattgoswami

[hardmaru @hardmaru]

Instead of forcing models to hold everything in an active context window, we can use hypernetworks to instantly compile documents and tasks directly into the model's weights. A step towards giving language models durable memory and fast adaptation.

Blog: https://t.co/iHoifpsLMu

[Andrej Karpathy @karpathy]

Cool chart showing the ratio of Tab complete requests to Agent requests in Cursor. With improving capability, every point in time has an optimal setup that keeps changing and evolving and the community average tracks the point. None -> Tab -> Agent -> Parallel agents -> Agent Teams (?) -> ???

If you're too conservative, you're leaving leverage on the table. If you're too aggressive, you're net creating more chaos than doing useful work.

The art of the process is spending 80% of the time getting work done in the setup you're comfortable with and that actually works, and 20% exploration of what might be the next step up even if it doesn't work yet.

[Andrej Karpathy @karpathy]

I had the same thought so I've been playing with it in nanochat. E.g. here's 8 agents (4 claude, 4 codex), with 1 GPU each running nanochat experiments (trying to delete logit softcap without regression). The TLDR is that it doesn't work and it's a mess... but it's still very pretty to look at :)

I tried a few setups: 8 independent solo researchers, 1 chief scientist giving work to 8 junior researchers, etc. Each research program is a git branch, each scientist forks it into a feature branch, git worktrees for isolation, simple files for comms, skip Docker/VMs for simplicity atm (I find that instructions are enough to prevent interference). Research org runs in tmux window grids of interactive sessions (like Teams) so that it's pretty to look at, see their individual work, and "take over" if needed, i.e. no -p.

But ok the reason it doesn't work so far is that the agents' ideas are just pretty bad out of the box, even at highest intelligence. They don't think carefully though experiment design, they run a bit non-sensical variations, they don't create strong baselines and ablate things properly, they don't carefully control for runtime or flops. (just as an example, an agent yesterday "discovered" that increasing the hidden size of the network improves the validation loss, which is a totally spurious result given that a bigger network will have a lower validation loss in the infinite data regime, but then it also trains for a lot longer, it's not clear why I had to come in to point that out). They are very good at implementing any given well-scoped and described idea but they don't creatively generate them.

But the goal is that you are now programming an organization (e.g. a "research org") and its individual agents, so the "source code" is the collection of prompts, skills, tools, etc. and processes that make it up. E.g. a daily standup in the morning is now part of the "org code". And optimizing nanochat pretraining is just one of the many tasks (almost like an eval). Then - given an arbitrary task, how quickly does your research org generate progress on it?

[OpenAI @OpenAI]

We’re sharing two papers on red teaming, which is an essential part of testing frontier AI models—a white paper on our approach for engaging external red teamers & a research study introducing a new automated red teaming method. https://t.co/JBS6hmRfw4

[General Intelligence @URNArtist_ai]

AI inference is compute intensive & energy is the bottle neck at the moment.

https://t.co/SkLlg2W5Y0

[Brett Adcock @adcock_brett]

It was a huge week of AI and robotics news.

So I summarized everything announced by OpenAI, Apple, Google DeepMind, Adobe, The White House, Mistral, Tencent, Runway, and more.

Here's everything you need to know and how to make sense out of it:

[Prakash @8teAPi]

No you don’t understand

> o1 reasoning engine -> gets better with reinforcement training
> underlies SearchGPT which then becomes a self-improving search engine reasoning through every query with data from the web
> then has access to share screen from your desktop so that it can watch what you do
> then has access to multimodality so that it can talk, generate images and videos
> then starts to reason about how to control your desktop
> meanwhile talking to you over your Apple phone
> starts generating little UI pieces when it needs to, buttons, graphs
> memory gets better, you start using it as Notepad, calculator, calendar etc.

The pieces are all there… but scaling and fixing the issues and debugging and latency and putting it into a product and distributing and making money from it are not.

But they will be.

[Prakash @8teAPi]

Why is OpenAI’s o1 a big deal?

> so it does better on some benchmarks so what?

Let me explain

> is new knowledge new? Ie
> or is it a combination of old knowledge and new data?
> that’s the crux of the debate

This is an old debate
> Kuhn’s scientific paradigm shifts for eg moving earth centric to sun centric model of solar system during Galileo was due to new theoretical model then verified by observation
> Vs Popper’s falsification, ie starting with a base of logic and experimentation you can find out anything

o1 is a Popperian falsification machine
> it breaks problems down into parts
> creates hyphothesis
> tests them

You can now easily imagine distilling o1 into a perfect Popperian machine, and then reducing its knowledge base of facts to a bare minimum.

The perfect reasoning engine working from first principles.

Of course that’s a dream, but we seem to be on the cusp of it.

[Noam Brown @polynoamial]

Today, I’m excited to share with you all the fruit of our effort at @OpenAI to create AI models capable of truly general reasoning: OpenAI's new o1 model series! (aka 🍓) Let me explain 🧵 1/

[Alex Kaplan @alexkaplan0]

Huge news today. This is the ChatGPT moment for drug discovery.

Chai-1 is a new open source foundation model that outperforms AlphaFold3 and ESM3 for molecular structure prediction.

Basically, paste sequence -> make drug. Check this out: (1/6)

[General Intelligence @URNArtist_ai]

https://t.co/pKmcZ1NiCq

[Emad @EMostaque]

Will be a lot more AIs in the future than humans.

[Just Jeff @japarjam]

Toto we aren’t in Kansas anymore. Talking SVM API and Moar! With @realbuffalojoe from @anza_xyz

[General Intelligence @URNArtist_ai]

https://t.co/YnVR9CkFdw

[Emad @EMostaque]

How can you not think generative AI disrupts the media sector looking at this.

Total generation time for each is likely a couple minutes, will get way faster & there will be complete control over all elements & consistency.

Just hasn’t all been pulled together yet.

[Turbin3 @solanaturbine]

This work was completed by @__lostin__ as his thesis project for our Advanced Research program. He and his peers deserve the credit for this work.

[Just Jeff @japarjam]

If only there was a group capable of training professionals on all these, who offered a program to top engineers and devs? If only this group had a program
Doing just that with ~30 highly qualified people in it already. Oh wait @solanaturbine is doing just that. @solana

[General Intelligence @URNArtist_ai]

@EMostaque https://t.co/LW45VxGL78

[Aakash Kumar Nain @A_K_Nain]

I went through the Llama-3 technical report (92 pages!). The report is very detailed, and it will be hard to describe everything in a single tweet, but I will try to summarize it in the best possible way. Here we go...

Overview
- Standard dense Transformer with minor changes
- No Mixture of Experts in favor of stable training
- Similarly, instead of RL in post-training, they use SFT, rejection sampling, and DPO

- Multimodal combining all three modalities: Text, Images, and speech. Multimodal models are still under active development

- Multimodal pre-training includes separate encoders for images and speech

-Image encoder pre-trained on image-text pairs, while speech encoder is pre-trained in a self-supervised fashion with masked inputs reconstructed via a discrete-token representation

-Both pre-trained encoders (image and speech) are connected to the pre-trained LM via vision and speech adapters respectively.

Language Model

1. Pre-Training

- 405B parameters
-15.6T tokens
- knowledge cutoff end of 2023

- context window of 8K tokens
- context window increased to 128K in the continued pre-training stage

- custom parser for HTML that maintains the structure of mathematics and code content
- deduplication performed on URL-level, document-level, and line-level

- n-gram coverage ratio to remove redundant text
dirty-word counting to remove adult content
- token-distribution KL divergence to filter documents with excessive outlier tokens
- fasttext-based model to classify documents into 176 languages.

- data ratio: 50% of tokens corresponding to general knowledge, 25% mathematical and reasoning tokens, 17% code tokens, and 8% multilingual tokens.

2. Model Architecture
- Model architecture same as llama and llama-2 with a few modifications (proving once again that quality data remains the king!)

- GQA with 8 KV heads
- RoPE frequency increased to 500,000

- attention mask that prevents self-attention between different documents within the same sequence

- 128K vocab size (100K from tiktoken, 28K additional tokens for languages other than English)

- 126 layers, 128 attention heads, and 16,384 embedding size

3. Infra and Scaling
- Llama 3 405B trained on up to 16K H100 GPUs, each running at 700W TDP with 80GB HBM3 using Meta’s Grand Teton AI server platform

- Tectonic (Meta's iternal) distributed file system for storage, 240PB, 7500 serves with SSDs, throughout ranging from 2TB/s to 7TB/s

- RoCE-based AI cluster comprises 24K GPUs connected by a three-layer network

- Parallelism, and schedulers optimized for the hardware topology

- Enhanced ECMP routing and deep-buffer switches for congestion control

- 4D parallelism: a combination of four different types of parallelism methods including tensor parallelism, pipeline parallelism, context parallelism, and data parallelism to shard the model

- In context parallelism, partitions are across the sequence dimension. Based on all-gather where they gather all keys and values, and then compute attention output for the local query tensor chunk

- Optimized order of parallelism for better network bandwidth and latency: TP, CP, PP, DP

- BF16 MFU of 38%-43%
- Gradient accumulation with FP32. For immediate tensors that are used in multiple places, like vision encoder output, the gradients are accumulated in FP32

-54-day snapshot pre-training with 467 interruptions. GPU issues account for 58% of the total issues (this is why TPUs are superior)

-During the day, the throughput of GPUs varies from 1-2% because of higher temperature

4. Training Recipe
- Cosine schedule, peak lr 8e-5, 8000 linear warmup steps, and decay to 8e-7 with 1,200,000 training steps.

- Initial batch size 4M tokens with sequence length 4096

- After training for 252M tokens, batch size was adjusted to 8M tokens with a sequence length of 8192
The batch size is doubled again to 16M tokens after training for 2.87T tokens

- Context length increased only when model performance on short-context evaluations has recovered completely, and the model perfectly solves “needle in a haystack” tasks up to that length.

- A total of six stages of gradual increment in context length
- Long context pretraining done with 800B training tokens.

5. Post-training
- SFT followed by DPO

- A mix of annotated and synthetic datasets (mostly generated data)

- First, train a reward model on human-annotated preference data followed by SFT and DPO

- New capabilities like tool use with a new multi-message chat protocol which uses various special header and termination tokens.

6. Reward Modeling
- For reward modeling, the training objective is the same except for the removal of the margin term in the loss because of diminishing return

- The reward model is used for rejection sampling on human-annotated prompts

- For rejection sampling, K (ranging from 10-30) outputs are chosen.
- PagedAttention is used to implement efficient rejection sampling

-Each preference ranking sample has two or three responses with rankings as: edited > chosen > rejected

7. Supervised finetuning
- SFT on rejection-sampled data combined with real data and synthetic data
- Prompt tokens masked (finally!)
- Trained with a lr of 1e-5 for 8.5K to 9K steps

8. Direct Preference Optimization
-trained on preference data collected using the best-performing models from the previous alignment rounds
lr=1e-5 and β=0.1

- Special tokens like header and termination tokens are masked out to stabilize the training. The presence of these tokens, both in the accepted and the rejected responses, leads to a conflicting learning objective.

- An additional negative log-likelihood (NLL) loss term with a scaling coefficient of 0.2  is applied to the chosen sequence.

----------------------------------------------------
Vision

1. Data
- Image-text and video-text pairs for the image encoder

- Annealing dataset created by resampling the image-caption pairs to a smaller volume of approx 350M examples using n-grams.

- Additional 150M examples collected using visual grounding, screenshot parsing, question-answer pair, synthetic captions, synthetically generated images from charts, tables, equations, etc represented via LaTex or markdown

- Video duration varies from 16-21 seconds, resolution varies from 320p to 4K

2. Model
- Three components: Image encoder, image adapter, and video adapter

Image encoder
- ViT/H-14 variant, 630M params, trained on 2.5B image-text pairs for five epochs. Image size is 224x224 split into 16x16 patches

- Multi-layer feature extraction from previous layers and injected into final layer to preserve fine-grained localization information

- 40 transformer blocks with 8 gated attention layers

Image Adapter
- GQA attention
- The cross-attention layer alone has almost 100B params (wt..😱🫤😵‍💫🫨)
- Trained in two stages

Video Adapter
- 64 frames uniformly sampled from a video each processed by image encoder
- Temporal information using temporal aggregator (perceiver resampler), and some additional cross-attention layers

3. Pre-training
- For image, started with pre-trained text model and vision encoder weights.

- Vision encoder is unfrozen, text frozen, and trained using 6B image-text pairs with a batch size of 16,384, cosine schedule, lr 10e-4 with weight decay of 0.01

- For video, start from the image pre-trained and annealed weights. Video aggregator trained from scratch while everything else is frozen

4. Post training
- Data procedures are mostly the same as in the case of text
- Academic datasets, human-annotated and synthetic data
- For quality tuning, a small but highly selective with utmost quality data is curated. DPO is done on this data to improve response quality to help improve human evaluations

---------------------------------------------------------
Speech

1. Data
- 15M hours of multilingual speech data for pretraining
- ASR training data contains 230K hours of manually transcribed
- speech recordings that span 34 languages.AST training data contains 90K hours of translations in two directions (33 languages -> English and English ->33 language)
- 25K hours of synthetic data

2. Model
- Speech encoder and speech adapter
- Speech encoder is Conformer model with 1B params.

- Input to the model is a 80 dimensional mel-spectrogram processed by 4-stride stacking layers followed by a linear projection before passing to the conformer encoder

- Each Conformer layer has a latent dimension of 1536,
and consists of two Macron-net style feed-forward networks with dimension 4096, a convolution module with kernel size 7, and a rotary attention module

- The speech adapter OTOH contains about 100M parameters. It is composed of a convolution layer, a rotary Transformer layer, and a linear layer.

3. Training
- Pre-training leverages unlabeled data to train a speech encoder

- Self-supervised BEST-RQ algorithm use to pre-train the encoder

- a mask of 32-frame length with a probability of 2.5% is applied to the input mel-spectrogram

- mel-spectrogram features are quantizied by stacking 4 consecutive frames, projecting the 320-dimensional vectors to a 16-dimensional space, and performing a NN search with cosine similarity metric within a codebook of 8,192 vectors

- 16 different codebooks
- multi-softmax loss is used only on masked frames for efficiency reasons.

- encoder trained for 500K steps with a global batch size of 2,048 utterances.

- In the second stage, SFT is done where the adapter and pre-trained encoder are integrated with the language model and trained jointly with it while the LLM stays frozen.

---------------------------------------------------
Results

---------------------------------------------------
Conclusion

Llama-3 is an extremely good model, and the technical report is pure gold. This is the kind of report you expect from people doing serious stuff. Other labs should take a note!

Also, it is pretty evident that not much is happening on the modeling side. The architecture is pretty much the same (and still surprises me that we have come this far).

In my honest opinion, more than the modeling capabilities, Llama-3 is an engineering marvel. So, if you are competing with a big lab, ask yourself "Do I have the infra to run things at this scale reliably and efficiently?"

Kudos to Zuck and the rest of the Meta team for doing this. You did what we expected from others to do from the beginning. Full support! ♥️🍻

[Andrew Curran @AndrewCurran_]

This morning five senate Democrats sent a letter to Sam Altman with twelve questions. This was question nine:

'Will OpenAI commit to making its next foundation model available to U.S. Government agencies for pre-deployment testing, review, analysis, and assessment?'