Toni Rodriguez

Researching LLMs

[Max Tena @MaxTena1]

Solo recordarles que mientas en los últimos 30 años, en la OCDE el salario real ha avanzado un 30,9% y en Estados Unidos un 47,5%, en España solo han subido un 2.8%. Sigan discutiendo que si izquierda y derecha, que si hombres y mujeres, que si catalanes y madrileños…

[Toni Rodriguez @tonirodriguez]

@v_sapronov @MalosPanes @acolombiadev https://t.co/qaneekvXSk

[Toni Rodriguez @tonirodriguez]

I've completed Technical AI Safety Project with BlueDot Impact! https://t.co/k3gF4gtVFB

[Aleksa Gordić (水平问题) @gordic_aleksa]

OpenAI's GPT-4 details have apparently been leaked! Looks very detailed and I suspect it's the real deal - given all I know about how these systems work. Here is a summary (extractive+abstractive) I made based on the original thread (see bottom of the post) + some additional pointers from me. (Disclaimer there are probably errors here)

---ARCH---
* GPT-4 is more than 10x the size of GPT-3 (175 B). We believe it has a total of ~1.8 trillion parameters across 120 layers. Mixture Of Experts (16 experts, each ~111B). Not a dense transformer like e.g. PaLM (or GPT-3). They use MQA instead of MHA (classic at this point).

* Each forward pass (generation of 1 token) only utilizes ~280B parameters and ~560 TFLOPs. This contrasts with the ~1.8 trillion parameters and ~3,700 TFLOP that would be required per forward pass of a purely dense model.

---DISTRIBUTED---

* To parallelize across all their A100s GPUs They utilized 8-way tensor parallelism. Beyond that, they are using 15-way pipeline parallelism. Also apparently they used DeepSpeed ZeRo Stage 1 or block-level FSDP.

(You can check out my video on all of these strategies here: https://t.co/FN4b63WHfk 3D parallelism is what you're looking for & ZeRo)

---VISION---

They have a separate vision encoder from the text encoder, with cross-attention. The architecture is similar to Google DeepMind's Flamingo (I used to work on this project :) ). This adds more parameters on top of the 1.8T of GPT-4. It is fine-tuned with another ~2 trillion tokens, after the text-only pre-training.

---DATA---

* Trained on ~13T tokens (multiple epochs, not unique). Plus millions of rows of instruction fine-tuning data from ScaleAI & internally (I guess acquired through ChatGPT + their API before they changed the policy).

* 8k context length for the pre-training phase. The 32k seqlen version of GPT-4 is based on fine-tuning of the 8k after the pre-training. See e.g. MosaicML's blog on how to achieve this: https://t.co/J0UfPVYFMQ)

---COST---

* OpenAI’s training FLOPS for GPT-4 is ~2.15e25, on ~25,000 A100s for 90 to 100 days at about 32% to 36% MFU. Part of this extremely low utilization is due to an absurd number of failures requiring checkpoints that needed to be restarted from.

If their cost in the cloud was about $1 per A100 hour, the training costs for this run alone would be about $63 million.

(Today, the pre-training could be done with ~8,192 H100 in ~55 days for $21.5 million at $2 per H100 hour)

---INFERENCE---

OpenAI might be using speculative decoding on GPT-4's inference. See this paper: https://t.co/nYWovzSb4I

The inference runs on a cluster of 128 GPUs. There are multiple of these clusters in multiple datacenters in different locations (it'll be hard for Elizier to nuke these xD). 8-way tensor parallelism and 16-way pipeline parallelism.

Original thread: https://t.co/Kt0UP0WlCl (strictly speaking the original one has been removed).

[Toni Rodriguez @tonirodriguez]

Curioso, en mi MacBook Pro esta, he ejecutado el comando y me ha abierto el paper de Satoshi 🫢🫢🫢🫢🤨

[MAD Cripto 💎🫱.og @mattunchi]

Hay una copia de el whitepaper de #bitcoin en todas las MacBook.

Apple es controlado por Satoshi? 👀

Si pones en terminal este comando, te aparece el whitepaper en tu mac:

open /System/Library/Image\ Capture/Devices/VirtualScanner.app/Contents/Resources/simpledoc.pdf

[Martín Bianchi Tasso @martinbianchi]

“La misión de un Papa no es estar con los convencidos”. Amén @jordievole https://t.co/5rsqrvnHAd vía @el_pais

[Héctor Mohedano @HMohedano_]

🍿 Para CELEBRAR la entrada de año 2023, aquí os dejo un LISTADO con las 7 PELÍCULAS sobre BOLSA y MERCADOS FINANCIEROS que todo INVERSOR debería ver al menos una vez.

Alguna quizá te SORPRENDA...

[Ivan Landabaso @IvanLandabaso]

Spain FTW

[Ben Tossell @bentossell]

AI tools that didn't exist one year ago:

ChatGPT
Whisper
GPT-3
Codex
GitHub Copilot
InstructGPT
Text-to-product
AI slides
DALLE + API
Midjourney
Stable Diffusion
Runway videos
Email AI
AI chrome extensions
Replit Ghostwriter
No-code AI app builders
Ben's bites 😉

what else?

[Toni Rodriguez @tonirodriguez]

@albertsampietro @esrafiki @vchamorr Un visionari com sempre ;-)

[retrochenta @retrochenta]

Hoy hace 50 años que empezó a emitirse en Japón la mítica serie "Mazinger Z". #mazingerz50

[Prof. Feynman @ProfFeynman]

Schrödinger's Plates: They are both broken and not broken until you open the door.

[Dani Sanchez-Crespo @DaniNovarama]

Bueno, pues ya tenemos la primera imagen del Telescopio Espacial Webb. Se la dejo aquí, si clican el enlace les baja la versión de resolución completa. Pero como siempre aquí lo interesante no es la foto, si no el "por qué" es interesante. Dentro hilo.

[John W. Rich (Wealthy) @Cokedupoptions]

Last year, I put my entire son's college fund in Ethereum

I purchased YouTube Premium with the remaining funds so he can just learn everything online

[THE OBJECTIVE @TheObjective_es]

Calviño anuncia la creación de la Agencia de Supervisión de Inteligencia Artificial.

Más noticias en nuestra web: https://t.co/P6YrYvQEWm

[Mark Tenenholtz @marktenenholtz]

You’re often told (and I’m guilty of this):

1. Do EDA
2. Do feature engineering
3. Fit a model

What’s wrong with this?

Fitting a model can tell you a lot about your data. EDA is 10x better when you a have a model telling you where to look

[Pedro Domingos @pmddomingos]

Which subfield of AI currently perceived as irrelevant will be the next big thing?