fairseq

Sequence modeling toolkit for @PyTorch

[Mikel Artetxe @artetxem]

We are releasing a family of dense and MoE language models with up to 13B and 1.1T parameters. We find that MoEs are more efficient, but the gap narrows at scale and varies greatly across domains and tasks.

Paper: https://t.co/3SeaCf2JjN

Models & code: https://t.co/2Rl6RT6Nkq

[fairseq @fairseq]

Models and code available in fairseq: https://t.co/ft8TxOSxDm

[Xian Li @xl_nlp]

🌍Few-shot learning beyond English🌏
📢 Announcing XGLMs, a series of multilingual autoregressive languages models setting new SoTA on few-shot learning and outperforming English-centric models (e.g. GPT-3).

Paper: https://t.co/nn4YD5bwFd
Models and code: https://t.co/ob0HkhnAP4

[fairseq @fairseq]

Mixture of experts training in fairseq is now 40% faster thanks to Microsoft's Tutel library!
Blog: https://t.co/9x9zG9HpwU
Fairseq code: https://t.co/lqZedr8HnT
Tutel code: https://t.co/SMqbjaVeVB

[AI at Meta @AIatMeta]

We’re introducing GSLM, the first language model that breaks free completely of the dependence on text for training. This “textless NLP” approach learns to generate expressive speech using only raw audio recordings as input. Learn more and get the code:
https://t.co/kRkUaFyZWb

[fairseq @fairseq]

fairseq now supports CPU offloading and full parameter+optimizer state sharding via fairscale's FullyShardedDataParallel module. See our tutorial to train a 13B parameter LM on 1 GPU: https://t.co/Zi65xy3NZw

[fairseq @fairseq]

We just released 0.10.0, which is our last significant release before 1.0.0 when we will migrate to @Hydra_Framework. Changelog: https://t.co/B8rTqdnwQO

[Naman Goyal @NamanGoyal21]

Facebook AI Research's sequence modeling library @fairseq has made it's twitter debut. Please follow for latest updates.

[PyTorch @PyTorch]

Fairseq includes support for sequence to sequence learning for speech and audio recognition tasks, faster exploration and prototyping of new research ideas while offering a clear path to production. https://t.co/luTz894A4y

[PyTorch @PyTorch]

roberta = torch.hub.load('pytorch/fairseq', 'roberta.large')

[PyTorch @PyTorch]

fairseq now supports the training of gated convolutional language models (https://t.co/Zo8J0lVTA1). It can train a Google Billion Word language model on 128 GPUs in less than a day.

[PyTorch @PyTorch]

FairSeq Toolkit - Major Update
- Distributed Training
- Transformer models (big Transformer on WMT Eng-German in < 5 hours on DGX-1)
- Fast Inference: translations @ 92 sent/sec for big Transformer
- Story Generation
Read more at Michael Auli's post: https://t.co/eptKDuh0WI

[Yann LeCun @ylecun]

Fairseq, now in PyTorch!
The open-source convolutional sequence-to-sequence engine from FAIR is now available in... https://t.co/E5tCqHV3eZ