Karthick

Data science consultant at MHP - Porsche Digital Lab, Berlin.

[Shilpi Yadav @ShilpiYlucknow]

@TeamCRonaldo @grok what a body?

[Andrej Karpathy @karpathy]

Congrats to @AIatMeta on Llama 3 release!! 🎉
https://t.co/UBwFPTJM6V
Notes:

Releasing 8B and 70B (both base and finetuned) models, strong-performing in their model class (but we'll see when the rankings come in @ @lmsysorg :))
400B is still training, but already encroaching GPT-4 territory (e.g. 84.8 MMLU vs. 86.5 4Turbo).

Tokenizer: number of tokens was 4X'd from 32K (Llama 2) -> 128K (Llama 3). With more tokens you can compress sequences more in length, cites 15% fewer tokens, and see better downstream performance.

Architecture: no major changes from the Llama 2. In Llama 2 only the bigger models used Grouped Query Attention (GQA), but now all models do, including the smallest 8B model. This is a parameter sharing scheme for the keys/values in the Attention, which reduces the size of the KV cache during inference. This is a good, welcome, complexity reducing fix and optimization.

Sequence length: the maximum number of tokens in the context window was bumped up to 8192 from 4096 (Llama 2) and 2048 (Llama 1). This bump is welcome, but quite small w.r.t. modern standards (e.g. GPT-4 is 128K) and I think many people were hoping for more on this axis. May come as a finetune later (?).

Training data. Llama 2 was trained on 2 trillion tokens, Llama 3 was bumped to 15T training dataset, including a lot of attention that went to quality, 4X more code tokens, and 5% non-en tokens over 30 languages. (5% is fairly low w.r.t. non-en:en mix, so certainly this is a mostly English model, but it's quite nice that it is > 0).

Scaling laws. Very notably, 15T is a very very large dataset to train with for a model as "small" as 8B parameters, and this is not normally done and is new and very welcome. The Chinchilla "compute optimal" point for an 8B model would be train it for ~200B tokens. (if you were only interested to get the most "bang-for-the-buck" w.r.t. model performance at that size). So this is training ~75X beyond that point, which is unusual but personally, I think extremely welcome. Because we all get a very capable model that is very small, easy to work with and inference. Meta mentions that even at this point, the model doesn't seem to be "converging" in a standard sense. In other words, the LLMs we work with all the time are significantly undertrained by a factor of maybe 100-1000X or more, nowhere near their point of convergence. Actually, I really hope people carry forward the trend and start training and releasing even more long-trained, even smaller models.

Systems. Llama 3 is cited as trained with 16K GPUs at observed throughput of 400 TFLOPS. It's not mentioned but I'm assuming these are H100s at fp16, which clock in at 1,979 TFLOPS in NVIDIA marketing materials. But we all know their tiny asterisk (*with sparsity) is doing a lot of work, and really you want to divide this number by 2 to get the real TFLOPS of ~990. Why is sparsity counting as FLOPS? Anyway, focus Andrej. So 400/990 ~= 40% utilization, not too bad at all across that many GPUs! A lot of really solid engineering is required to get here at that scale.

TLDR: Super welcome, Llama 3 is a very capable looking model release from Meta. Sticking to fundamentals, spending a lot of quality time on solid systems and data work, exploring the limits of long-training models. Also very excited for the 400B model, which could be the first GPT-4 grade open source release. I think many people will ask for more context length.

Personal ask: I think I'm not alone to say that I'd also love much smaller models than 8B, for educational work, and for (unit) testing, and maybe for embedded applications etc. Ideally at ~100M and ~1B scale.

Talk to it at https://t.co/KmKRlZeTHQ
Integration with https://t.co/RD6MRWT2zz

[Sundar Pichai @sundarpichai]

Seeing some qs on what Gemini *is* (beyond the zodiac :). Best way to understand Gemini’s underlying amazing capabilities is to see them in action, take a look ⬇️

[Stability AI @StabilityAI]

Today, we are proud to join the AI Alliance, a significant initiative that aligns with our commitment to amplify human intelligence by making foundational AI technology accessible to all.

The formation of this global collective brings together leaders across academia, research, science and industry – including @IBM, @intel, @huggingface, @RedHat, and many more – to advance AI that is open, innovative and responsible.

We invite you to learn more about our efforts to further this mission: https://t.co/1SLwXi1zXQ

[Jakub Beránek @Beranek1582]

Wrote down some thoughts about applying strong typing and "soundness" concepts from Rust when writing Python code
https://t.co/spzD8mcRnE

[Karthick @phykar]

@DVCorg Hi

[🦉DVC @DVCorg]

We made in-depth research and collated the Best MLOps resources, We want to help over 500 people get certified in MLOps before April 30th.

You will learn about:

- MLOps.
- Notebook to production etc

Retweet and Comment "Hi " or "I want" and we will share you in your DM.

[Emad @EMostaque]

Delighted to announce the public open source release of #StableDiffusion!

Please see our release post and retweet! https://t.co/dEsBX7cRHw

Proud of everyone involved in releasing this tech that is the first of a series of models to activate the creative potential of humanity

[Mike Tamir, PhD @MikeTamir]

Advances in Understanding, Improving, and Applying Contrastive Learning https://t.co/Ut8ajHkpMz
#AI #DataScience #DeepLearning #MachineLearning

[Jeremy Howard @jeremyphoward]

Really excited to share the amazing data and insights of @wightmanr with you all, regarding how to choose an image model.

I've created a @kaggle notebook, which rebuilds every day with latest data, & shows which models are the fastest and most accurate
https://t.co/kK37ieZHgN

[Marc Skov Madsen, PhD, CFA® @MarcSkovMadsen]

Want to know what makes @Panel_org especially powerful for building #dataapps?

Check out my blog post https://t.co/X0utPuR0VM

#python #dataviz #businessintelligence #analytics #datascience

[Karthick @phykar]

Google AI Blog:Nested Hierarchical Transformer: Towards Accurate, Data-Efficient, and Interpretable Visual Understanding https://t.co/bLUrGhAqfv via @googleai

[Karthick @phykar]

DeepMind: Competitive programming with AlphaCode https://t.co/BY9orfbH7N

[Karthick @phykar]

@manesh_mahdi Here is what Linus pushed (line 6) https://t.co/JtvCZJOHCE

[Elie Bursztein @elie]

🎊🥳TensorFlow Similarity 0.15 is out and it's a big one ! It adds all you need to train self-supervised models with STOA algorithms. To get started head out to the hello world notebook → https://t.co/CQW58kHon6

#TensorFlow #DeepLearning @TensorFlow @fchollet @JeffDean

[Karthick @phykar]

Google AI Blog:Interpretable Deep Learning for Time Series Forecasting https://t.co/BVJZVODyKr via @googleai

[Jeff Boudier 🤗 @jeffboudier]

Today @awscloud launched SageMaker Studio Lab to make ML more accessible, it's been fun collaborating toward the launch and lovely to see some Hugs Inside! 😍
https://t.co/kV90Dao1Wv

[Karthick @phykar]

Check out this #Python tutorial: Python 3.10: Cool New Features for You to Try by @realpython https://t.co/lgYtRfVhIC

[Karthick @phykar]

Introducing Pedalboard: Spotify’s Audio Effects Library for Python   https://t.co/Vn14wnARMD

[Karthick @phykar]

Introducing Pedalboard: Spotify’s Audio Effects Library for Python   https://t.co/Vn14wnARMD