Peter Humphreys

AI, quantum and neuroscience. Scientist @ DeepMind

[Peter Humphreys @p_humphreys]

Standard transformer-based language models use the same amount of compute for each token. Our new method, which we call Mixture-of-Depths, allows transformers to instead learn to dynamically allocate compute to specific positions in a sequence.

https://t.co/g6FD5EAXGm

[George Grigorev @iamgrigorev]

I have implemented Mixture-of-Depths and it shows significant memory reduction during training and 10% speed increase. I will verify if it achieves the same quality with 12.5% active tokens.
https://t.co/wbOByZZz4o
thanks @haeggee for initial code

[Piotr Padlewski @PiotrPadlewski]

Big fan of the work of @santoroAI and others. Glad Google decided to finally release it!
https://t.co/paO9sawurl

[Alex Hägele @haeggee]

... and with one experiment, I was able to roughly reproduce their results for a ~220M GPT-2. It gives a speedup of ~20min (80min dense vs 60min MoD, 4 A100s) while keeping the pplx close. This roughly matches Fig. 3 or 4 in the paper https://t.co/bF69KjPBWy

[Adam Santoro @santoroAI]

Transformers can be made sparse across their depth. When trained isoFLOP, we can match or exceed the performance of vanilla models, while saving inference FLOPs

https://t.co/jWl1wuHEko

[Peter Humphreys @p_humphreys]

We are excited to explore further methods of dynamically allocating compute to where it can be best used. Work with my lovely collaborators: David Raposo, @ritterstorm, @adamsantoro, @tyrell_turing, Timothy Lillicrap

[Peter Humphreys @p_humphreys]

Standard transformer-based language models use the same amount of compute for each token. Our new method, which we call Mixture-of-Depths, allows transformers to instead learn to dynamically allocate compute to specific positions in a sequence.

https://t.co/g6FD5EAXGm

[Peter Humphreys @p_humphreys]

Instead of processing all tokens (T) at each block of a transformer, we use learned routing similar to Mixture-of-Experts (MoE) to process k<T tokens instead. We can use this saved compute to train for longer or to introduce new parameters. We also save compute at inference time.

[PHD Comics @PHDcomics]

A new method for reviews https://t.co/gjg1mfrADW

[Peter Humphreys @p_humphreys]

@hannaballoo For the Best Falafel, Do It All Yourself http://t.co/EcWWk16uBm

[Peter Humphreys @p_humphreys]

@hannaballoo Tewittt tewoo!