Blog Post (w/ @gail_w): On "Thinking Like Transformers"
In which, I get a bit obsessed with learning how to code in Transformer lang🤖.
https://t.co/Nb6G52vuiK
(You can follow along or do the exercises yourself in a colab notebook.)
Regarding recent work on more biologically plausible "forward-only" backprop-like methods: in 2021, our VSML net already meta-learned backprop-like learning algorithms running solely in forward-mode - no hardwired derivative calculation! https://t.co/zAZGZcYtmO
🔥Our work has now been accepted to NeurIPS 2022 !!
`Toward a realistic model of speech processing in the brain with self-supervised learning’:
https://t.co/weiGlaiD65
Let’s meet in New Orleans on Tue 29 Nov 2:30pm PST (Hall J #524).
A recap of the 3 main results below 👇
Jonas was an awesome intern and here is the paper that proves it!
Multilingual models are known to suffer from the "curse of multilinguality", which causes performance to drop as they cover more languages. Our new model X-Mod addresses this using language-specific parameters.
Since Transformer LMs were invented, we’ve wanted them to be able to read longer inputs during inference than they saw during training. Our Attention with Linear Biases enables this, in very few lines of code, without requiring extra params or runtime https://t.co/GIhnSWvXnh 🧵⬇
FACT: #COVID19 is NOT airborne.
The #coronavirus is mainly transmitted through droplets generated when an infected person coughs, sneezes or speaks.
To protect yourself:
-keep 1m distance from others
-disinfect surfaces frequently
-wash/rub your 👐
-avoid touching your 👀👃👄
I Now call it "self-supervised learning", because "unsupervised" is both a loaded and confusing term.
In self-supervised learning, the system learns to predict part of its input from other parts of it input. In... https://t.co/GpcHEbj5jc