I've written up my study group lectures on implementing Transformers in PyTorch into a blog series:
Creating Transformers from Scratch:
- Part 1: The Attention Mechanism https://t.co/ZCMiCMzQIp
- Part 2: The Rest of the Transformer https://t.co/kWq9gWkvtp
[75min talk] i finally recorded this lecture I gave two weeks ago because people kept asking me for a video
so here it is, enjoy "The Little guide to building Large Language Models in 2024"
tried to keep it short and comprehensive – focusing on concepts that are crucial for training good LLM but often hidden in tech reports
🥁 Llama3 is out 🥁
8B and 70B models available today.
8k context length.
Trained with 15 trillion tokens on a custom-built 24k GPU cluster.
Great performance on various benchmarks, with Llam3-8B doing better than Llama2-70B in some cases.
More versions are coming over the next few months.
https://t.co/EkU9aIHdZE
Try out the NEW multimodal 🖐️hand+ 🎮controller interaction in Meta's interaction SDK as well as:
- 2D #UI panels manipulation and responsive #design
- Teleportation and locomotion with hand-tracking
- hand+body poses
- ... and a lot more I don't show in the video
As you know, my explorations of the Gen AI space is ultimately all about creative control. You should be able to shape the generative matter using all your artistic sensibilities and your aesthetic sense.
OpenAI's Sora is a huge technological leap, but what excites me the most about it is the modalities where it depends on input other than text alone. Such as video to video. Here's an example of how Sora can change an input video.
Base video🧵