mimmo iaia

MLOps, NLP Engineer, #AI Advocate .

[Andrej Karpathy @karpathy]

Congrats to @AIatMeta on Llama 3 release!! 🎉
https://t.co/UBwFPTJM6V
Notes:

Releasing 8B and 70B (both base and finetuned) models, strong-performing in their model class (but we'll see when the rankings come in @ @lmsysorg :))
400B is still training, but already encroaching GPT-4 territory (e.g. 84.8 MMLU vs. 86.5 4Turbo).

Tokenizer: number of tokens was 4X'd from 32K (Llama 2) -> 128K (Llama 3). With more tokens you can compress sequences more in length, cites 15% fewer tokens, and see better downstream performance.

Architecture: no major changes from the Llama 2. In Llama 2 only the bigger models used Grouped Query Attention (GQA), but now all models do, including the smallest 8B model. This is a parameter sharing scheme for the keys/values in the Attention, which reduces the size of the KV cache during inference. This is a good, welcome, complexity reducing fix and optimization.

Sequence length: the maximum number of tokens in the context window was bumped up to 8192 from 4096 (Llama 2) and 2048 (Llama 1). This bump is welcome, but quite small w.r.t. modern standards (e.g. GPT-4 is 128K) and I think many people were hoping for more on this axis. May come as a finetune later (?).

Training data. Llama 2 was trained on 2 trillion tokens, Llama 3 was bumped to 15T training dataset, including a lot of attention that went to quality, 4X more code tokens, and 5% non-en tokens over 30 languages. (5% is fairly low w.r.t. non-en:en mix, so certainly this is a mostly English model, but it's quite nice that it is > 0).

Scaling laws. Very notably, 15T is a very very large dataset to train with for a model as "small" as 8B parameters, and this is not normally done and is new and very welcome. The Chinchilla "compute optimal" point for an 8B model would be train it for ~200B tokens. (if you were only interested to get the most "bang-for-the-buck" w.r.t. model performance at that size). So this is training ~75X beyond that point, which is unusual but personally, I think extremely welcome. Because we all get a very capable model that is very small, easy to work with and inference. Meta mentions that even at this point, the model doesn't seem to be "converging" in a standard sense. In other words, the LLMs we work with all the time are significantly undertrained by a factor of maybe 100-1000X or more, nowhere near their point of convergence. Actually, I really hope people carry forward the trend and start training and releasing even more long-trained, even smaller models.

Systems. Llama 3 is cited as trained with 16K GPUs at observed throughput of 400 TFLOPS. It's not mentioned but I'm assuming these are H100s at fp16, which clock in at 1,979 TFLOPS in NVIDIA marketing materials. But we all know their tiny asterisk (*with sparsity) is doing a lot of work, and really you want to divide this number by 2 to get the real TFLOPS of ~990. Why is sparsity counting as FLOPS? Anyway, focus Andrej. So 400/990 ~= 40% utilization, not too bad at all across that many GPUs! A lot of really solid engineering is required to get here at that scale.

TLDR: Super welcome, Llama 3 is a very capable looking model release from Meta. Sticking to fundamentals, spending a lot of quality time on solid systems and data work, exploring the limits of long-training models. Also very excited for the 400B model, which could be the first GPT-4 grade open source release. I think many people will ask for more context length.

Personal ask: I think I'm not alone to say that I'd also love much smaller models than 8B, for educational work, and for (unit) testing, and maybe for embedded applications etc. Ideally at ~100M and ~1B scale.

Talk to it at https://t.co/KmKRlZeTHQ
Integration with https://t.co/RD6MRWT2zz

[Grok @grok]

@elonmusk @xai ░W░E░I░G░H░T░S░I░N░B░I░O░

[mimmo iaia @mimmoiaia]

A new version of Loquace is available!
Loquace is an Italian LLM based on Mistral, it's good at following instructions, responsive to prompt-engineering, RAG ready and trained for just about 1 euro! https://t.co/u3yqt2HU6b
#LLM #Italian #Opensource

[Mistral AI @MistralAI]

magnet:?xt=urn:btih:5546272da9065eddeb6fcd7ffddeef5b75be79a7&dn=mixtral-8x7b-32kseqlen&tr=udp%3A%2F%https://t.co/uV4WVdtpwZ%3A6969%2Fannounce&tr=http%3A%2F%https://t.co/g0m9cEUz0T%3A80%2Fannounce

RELEASE a6bbd9affe0c2725c1b7410d66833e24

[mimmo iaia @mimmoiaia]

This version is based on OpenLLaMA 13B and this tokenizer have actually learned *some* Italian words before, this proves that it is possible to fine-tune an LLM in a specific language if the base model have some basic knowledge of it.

[mimmo iaia @mimmoiaia]

The latest version of Loquace is pretty good at speaking but it should definitely study more.
L'ultima versione di Loquace è di certo brava a parlare ma dovrebbe decisamente studiare di più. #llm #italian

[Yann LeCun @ylecun]

This is huge: Llama-v2 is open source, with a license that authorizes commercial use!

This is going to change the landscape of the LLM market.
Llama-v2 is available on Microsoft Azure and will be available on AWS, Hugging Face and other providers

Pretrained and fine-tuned models are available with 7B, 13B and 70B parameters.

Llama-2 website: https://t.co/PKrrXgHdem
Llama-2 paper: https://t.co/aINNrXNhMb

A number of personalities from industry and academia have endorsed our open source approach: https://t.co/N7HwgW9Suh

[mimmo iaia @mimmoiaia]

Ho appena pubblicato il mio ultimo e divertente articolo su come addestrare una propria Intelligenza Artificiale "Chiacchierona", come Loquace, utilizzando pochissime risorse. Grazie ancora a Genesis Cloud per aver fornito i credit…https://t.co/LR20YnWl1e https://t.co/vV1yqqRG1J

[mimmo iaia @mimmoiaia]

I just published my trying-to-be-funny article about how to fine-tune a large language model using very little resources on your own data/language. Thanks once again Genesis Cloud for providing the cloud resources and the free cred…https://t.co/jEtlzFc4EO https://t.co/1ZRNcrvCBU

[mimmo iaia @mimmoiaia]

If anyone was thinking AI progress was slowing down this week: 256k Tokens of context on a Large Language Model.

Mind-blowing has become the new normal for LLMs!
#LLM #opensource https://t.co/IC5xZzOsBV

[mimmo iaia @mimmoiaia]

If anyone was thinking AI progress was slowing down: 256k Tokens of context on a Large Language Model.

#LLM #opensource https://t.co/IC5xZzOsBV

[mimmo iaia @mimmoiaia]

https://t.co/yIIvEyMxuN https://t.co/0h6q3mGXbX

[mimmo iaia @mimmoiaia]

What is hot right now? Super-hot is! 8k of context without fine-tuning.
#LLM #AI https://t.co/w5TiasBDxr

[kache @yacineMTB]

two lines of code is all you need for 8k context
https://t.co/6GqXMfvfmS

[choomba @beginanon]

Karpathy was so right about vector databases. It took me less than 100 loc to whip up something decent, and the memory usage is negligible.

np.array is all you need

[hardmaru @hardmaru]

OpenLLaMA-13B released‼️

OpenLLaMA aims to train Apache-licensed “drop in” replacement for Meta’s LLaMA models. They have been trained for 1T tokens on RedPajama dataset.

Given the popularity of models based on LLaMA-13B, this one should be quite useful.
https://t.co/YRrcI1bHiy

[mimmo iaia @mimmoiaia]

This is pretty awesome: #LLM https://t.co/IGubSFNL03

[mimmo iaia @mimmoiaia]

I'm happy to announce the first release of the Loquace Italian family, a suite of Large Language Models fine-tuned on Italian Instructions !!!
The Loquace models are created as a proof-of-concept to evaluate on how language tuning…https://t.co/RKdKuifn0o https://t.co/QMkoDK2fuP

[mimmo iaia @mimmoiaia]

Quantization is awesome. The things that are becoming possible every day are mind-blowing. LLMs are going to be a commodity VERY soon. https://t.co/yq3nTWZCbH

[mimmo iaia @mimmoiaia]

Loquace, la prima grande Intelligenza Artificiale che impara dall'italiano, ha appena finito il suo primo giorno di scuola... A presto le introduzioni ufficiali! #LLM # #intelligenzaartificiale #gpt #italiano #opensource #cc-by-nc #genesiscloud