Olivia
Online
Omega Gamage
@AlpKnight
Joined October 2022
12 Following
0 Followers
16 Posts
I just won $408.56 on @Polymarket!
Join me and put your money where your mouth is: https://t.co/zCxqF1gG3r
My take on the Post Pretraining world - Ilya’s talk:
Ilya is implying we need to find something else to scale - the brain–body mass ratio graph in the talk showed human intelligence “scaled” better than mammals.
LSTMs got out-scaled by transformers - the goal is to "edit" the scaling laws to make it more efficient.
Evolution somehow first tried scaling intelligence for mammals, then pushed the frontier up for non-human primates. Large elephants which exceeded the 700g gram wall were extinct in the end. Then hominids came along and broke the wall, and scaled far better. [0]
(A) Kaplan et al’s scaling laws [1] shows if we increase TRAINING compute = N (# parameters) * D (# tokens / data), the test loss also decreases in a log-log setting.
(A)* Instead of scaling TRAINING compute, Sutskever mentioned we can scale TEST TIME compute through search, or like O1 / QwQ etc.
(B) First on D (scaling data). There exists a theoretical “Data Wall” which is when all the data in the world (the internet and everything else) gets consumed by large models. Once we reach that point, we have to find ways to overcome this barrier to make models to continue to scale.
This could mean Synthetic Data Generation as Sutskever mentioned - literally using a trained model to augment datasets. The question is if this will plateau or keep scaling. Another approach is to make data scaling more efficient through better filtering. The FineWeb [2] dataset is one example of this.
We can also do more RL & post-training via DPO, PPO etc to squeeze more performance out of the same amount of tokens as explained in Lambert’s blog post [3]. These move the frontier downwards.
(C) Second on N (# of parameters) - the trick is to move to active parameters instead of total parameters. Large labs like OpenAI replaced MLP / FFNs in Dense transformers with MoE layers [4]. Instead of doing huge matrix multiplies, we smartly only select a few column groups to multiply instead, and leave the rest as 0. We can scale transformers to trillions of parameters like in Switch transformers [5].
(C)(i) Coincidentally Meta released multiple papers including one on Byte Latent Transformers [6] and Memory Layers [7]. BLTs edit the scaling laws itself by changing the definition of “tokens” in data scaling and also adding more to the non embedding parameters.
BLTs remove BPE tokenization by instead learning to allocate more optimum amounts of tokens / bytes to certain groups of patches through a smaller encoder. We then run a transformer on combined patches, and use a decoder for prediction.
(D) Memory Layers are what really interested me! They are essentially sparse lookup tables - first devised as Product Key layers in Lample et al’s paper [8] we replace the FFN MLP with a gigantic learnable matrix of size (100M, d) called V (Values). We then only select the top K rows of V (say 4) via a weighted sum via the softmax.
To find the top 4, we need another matrix K (Keys) of size (100M, d) to allow simple dot products to obtain the top indices. This essentially converts the dense MLP into a weighted sparse lookup table.
The issue is finding the top K rows needs 100M operations since we need to do (K * q) to obtain the indices. Accessing V is easy, and we can offload V to RAM. The trick in [8] is to use Fast Approximate Nearest Neighbors to find the top k rows. But this is hard to differentiate during training, so instead we do another trick - we split K (100M, d) into 2 matrices KA and KB both (sqrt(100M), d/2) in size, and use the Cartesian product.
(E) The Cartesian product of KA and KB is size (100M, d) - every row of KA (1, d/2) corresponds to the entire KB matrix (sqrt(100M), d/2), and since we have sqrt(100M) rows in KA, the total Cartesian product is of size sqrt(100M) * (sqrt(100M, d/2 + d/2) = (100M, d)
To get indices of 0 to N-1, we can then simply observe to find the largest dot product of (a^2 + b^2), we can find the max of (a^2) then the max of (b^2), and combine them separately. So the indices are simply sqrt(N) * topK_indices (KA * q) + topK_indices (KB * q).
This is super cool since we can now scale these sparse lookup tables to massive scales and only using a small (sqrt(100M), d) extra space. The [7] paper also adds a non linearity like in GLU [9] variants, and this is called the Memory+ layer, and this scales better than MoEs!
(F) A long post, but my final talk is Ilya is saying we need to find something else to scale. This could be:
1) Scaling instead test time via search, agents, O1.
2) Changing the arch by holding training compute constant like MoEs, Memory+ layers etc.
3) Changing the scales for scaling laws ie like BLTs.
4) Breaking the Data Wall via Synthetic Data Generation, RL, DPO, PPO, filtering etc.
5) Or something else!
I watched Ilya’s talk here: https://t.co/oE0AH0sp44
References:
[0] Brain–body mass ratio https://t.co/79wHIBijeK
[1] Kaplan et al “Scaling Laws for Neural Language Models” https://t.co/GrcnJ0gb4d
[2] Penedo et al “The FineWeb Datasets” https://t.co/uXqOxp8QNi
[3] Lambert RL for the masses https://t.co/vDXTANzQSU
[4] Shazeer et al “Outrageously Large Neural Networks” https://t.co/AFIMaGc90Y
[5] Fedus et al “Switch Transformers” https://t.co/YIfKHR5sQ9
[6] Pagnoni et al “Byte Latent Transformer” https://t.co/XFoqoRiwoV
[7] Berges et al “Memory Layers at Scale” https://t.co/g6SitSpHSo
[8] Lample et al “Large Memory Layers with Product Keys” https://t.co/Dyrs2rwDCx
[9] Shazeer “GLU Variants Improve Transformer” https://t.co/ooZ90oEEtS
![danielhanchen's tweet photo. My take on the Post Pretraining world - Ilya’s talk:
Ilya is implying we need to find something else to scale - the brain–body mass ratio graph in the talk showed human intelligence “scaled” better than mammals.
LSTMs got out-scaled by transformers - the goal is to "edit" the scaling laws to make it more efficient.
Evolution somehow first tried scaling intelligence for mammals, then pushed the frontier up for non-human primates. Large elephants which exceeded the 700g gram wall were extinct in the end. Then hominids came along and broke the wall, and scaled far better. [0]
(A) Kaplan et al’s scaling laws [1] shows if we increase TRAINING compute = N (# parameters) * D (# tokens / data), the test loss also decreases in a log-log setting.
(A)* Instead of scaling TRAINING compute, Sutskever mentioned we can scale TEST TIME compute through search, or like O1 / QwQ etc.
(B) First on D (scaling data). There exists a theoretical “Data Wall” which is when all the data in the world (the internet and everything else) gets consumed by large models. Once we reach that point, we have to find ways to overcome this barrier to make models to continue to scale.
This could mean Synthetic Data Generation as Sutskever mentioned - literally using a trained model to augment datasets. The question is if this will plateau or keep scaling. Another approach is to make data scaling more efficient through better filtering. The FineWeb [2] dataset is one example of this.
We can also do more RL & post-training via DPO, PPO etc to squeeze more performance out of the same amount of tokens as explained in Lambert’s blog post [3]. These move the frontier downwards.
(C) Second on N (# of parameters) - the trick is to move to active parameters instead of total parameters. Large labs like OpenAI replaced MLP / FFNs in Dense transformers with MoE layers [4]. Instead of doing huge matrix multiplies, we smartly only select a few column groups to multiply instead, and leave the rest as 0. We can scale transformers to trillions of parameters like in Switch transformers [5].
(C)(i) Coincidentally Meta released multiple papers including one on Byte Latent Transformers [6] and Memory Layers [7]. BLTs edit the scaling laws itself by changing the definition of “tokens” in data scaling and also adding more to the non embedding parameters.
BLTs remove BPE tokenization by instead learning to allocate more optimum amounts of tokens / bytes to certain groups of patches through a smaller encoder. We then run a transformer on combined patches, and use a decoder for prediction.
(D) Memory Layers are what really interested me! They are essentially sparse lookup tables - first devised as Product Key layers in Lample et al’s paper [8] we replace the FFN MLP with a gigantic learnable matrix of size (100M, d) called V (Values). We then only select the top K rows of V (say 4) via a weighted sum via the softmax.
To find the top 4, we need another matrix K (Keys) of size (100M, d) to allow simple dot products to obtain the top indices. This essentially converts the dense MLP into a weighted sparse lookup table.
The issue is finding the top K rows needs 100M operations since we need to do (K * q) to obtain the indices. Accessing V is easy, and we can offload V to RAM. The trick in [8] is to use Fast Approximate Nearest Neighbors to find the top k rows. But this is hard to differentiate during training, so instead we do another trick - we split K (100M, d) into 2 matrices KA and KB both (sqrt(100M), d/2) in size, and use the Cartesian product.
(E) The Cartesian product of KA and KB is size (100M, d) - every row of KA (1, d/2) corresponds to the entire KB matrix (sqrt(100M), d/2), and since we have sqrt(100M) rows in KA, the total Cartesian product is of size sqrt(100M) * (sqrt(100M, d/2 + d/2) = (100M, d)
To get indices of 0 to N-1, we can then simply observe to find the largest dot product of (a^2 + b^2), we can find the max of (a^2) then the max of (b^2), and combine them separately. So the indices are simply sqrt(N) * topK_indices (KA * q) + topK_indices (KB * q).
This is super cool since we can now scale these sparse lookup tables to massive scales and only using a small (sqrt(100M), d) extra space. The [7] paper also adds a non linearity like in GLU [9] variants, and this is called the Memory+ layer, and this scales better than MoEs!
(F) A long post, but my final talk is Ilya is saying we need to find something else to scale. This could be:
1) Scaling instead test time via search, agents, O1.
2) Changing the arch by holding training compute constant like MoEs, Memory+ layers etc.
3) Changing the scales for scaling laws ie like BLTs.
4) Breaking the Data Wall via Synthetic Data Generation, RL, DPO, PPO, filtering etc.
5) Or something else!
I watched Ilya’s talk here: https://t.co/oE0AH0sp44
References:
[0] Brain–body mass ratio https://t.co/79wHIBijeK
[1] Kaplan et al “Scaling Laws for Neural Language Models” https://t.co/GrcnJ0gb4d
[2] Penedo et al “The FineWeb Datasets” https://t.co/uXqOxp8QNi
[3] Lambert RL for the masses https://t.co/vDXTANzQSU
[4] Shazeer et al “Outrageously Large Neural Networks” https://t.co/AFIMaGc90Y
[5] Fedus et al “Switch Transformers” https://t.co/YIfKHR5sQ9
[6] Pagnoni et al “Byte Latent Transformer” https://t.co/XFoqoRiwoV
[7] Berges et al “Memory Layers at Scale” https://t.co/g6SitSpHSo
[8] Lample et al “Large Memory Layers with Product Keys” https://t.co/Dyrs2rwDCx
[9] Shazeer “GLU Variants Improve Transformer” https://t.co/ooZ90oEEtS](https://pbs.twimg.com/media/Ge8U7-lbwAIIiKV.jpg)
Best Reinforcement Learning course on YouTube
We made a step-by-step tutorial on how to finetune Llama-3 with Google Colab & deploy it to @Ollama
Tutorial: https://t.co/YtxXzlFOSp
Colab notebook: https://t.co/FnmHK9sysS
Blog post & video coming soon. 🦥
@SkySportsTennis Sky News, as always leader of fake news.
Announcing GPT-4, a large multimodal model, with our best-ever results on capabilities and alignment: https://t.co/TwLFssyALF
A challenge that every graduate student faces: how to read difficult research articles.
Here's how to understand research articles better using an AI-powered app (no cheating although it may feel like it 😀):
My favorite LLM: Large Lagrange Model...
We crossed 100,000 public AI models on the @huggingface hub available for free to all. Thank you to the whole community of contributors. Proud to make ML more open & collaborative!
Jupyter Lab Tip
If your notebook disconnects, try this ⬇️
Select👆 Kernel and choose Reconnect to Kernel
@jeremyphoward I think this could be useful https://t.co/CL1seYSQKy. It is referred in this book https://t.co/l1caNNDM3j
A100s are in high demand recently.
You can reserve A100's 80GB for a month at @jarvislabsai
Share it with your friends, please 😍
https://t.co/RgSWKvh2U5
Refer Friends.
Get 100 USDT Each. https://t.co/yrCNRPVo0D
Check out my portfolio on Shutterstock! https://t.co/hgGx75w5iP
Last Seen Users on Sotwe
Trends for you
Most Popular Users
Elon Musk 
@elonmusk
241.6M followers
Barack Obama
@barackobama
119M followers
Cristiano Ronaldo
@cristiano
114M followers
Donald J. Trump
@realdonaldtrump
111.8M followers
Narendra Modi
@narendramodi
107.2M followers
Rihanna
@rihanna
98.7M followers
NASA
@nasa
92.4M followers
Justin Bieber
@justinbieber
91.8M followers
KATY PERRY
@katyperry
89.8M followers
Taylor Swift
@taylorswift13
83.8M followers
Lady Gaga
@ladygaga
75.2M followers
Virat Kohli
@imvkohli
73M followers
Kim Kardashian
@kimkardashian
70.8M followers
YouTube
@youtube
68.8M followers
Neymar Jr
@neymarjr
66M followers
Bill Gates
@billgates
65M followers
Selena Gomez
@selenagomez
62.9M followers
The Ellen Show
@theellenshow
62.3M followers
CNN
@cnn
61.8M followers
X
@x
60.7M followers