BigScience Large Model Training

Follow the training of "BLOOM 🌸", the @BigScienceW multilingual 176B parameter open-science open-access language model, a research tool for the AI community.

[BigScience Large Model Training @BigScienceLLM]

The BLOOM model is now officially released!

Read more here: https://t.co/ZrwQVuM7DK
Find the model here: https://t.co/t7lAuaimNI

[BigScience Research Workshop @BigscienceW]

BLOOM is here. The largest open-access multilingual language model ever. Read more about it or get it at
https://t.co/mE013I62In
https://t.co/KrBRVklXLf

[clem 🤗 @ClementDelangue]

The Bloom paper is out. Looks like it's doing worse than current GPT3 API in zero-shot generation tasks in English but better than other open-source LLMs & better than all in zs multi-lingual (which was the main goal). Proud of the work from the community! https://t.co/NMHIzi1F79

[Niklas Muennighoff @Muennighoff]

Crosslingual Generalization through Multitask Finetuning 🌸

Demo: https://t.co/3ikMHdSEdY
📜 https://t.co/boEmB6BeYp
💻https://t.co/Ey8XtyUxgU

We present BLOOMZ & mT0, a family of models w/ up to 176B params that follow human instructions in >100 languages zero-shot. 1/7

[BigScience Large Model Training @BigScienceLLM]

The super-fast inference solutions are finally here for all to use:

[Stas Bekman @StasBekman]

Learn how you can get under 1msec per token generation time with BLOOM 176B model!

Not one, but multiple super-fast solutions including Deepspeed-Inference, Accelerate and Deepspeed-ZeRO!

https://t.co/mzOuKZ57W0

[clem 🤗 @ClementDelangue]

What do @StabilityAI @EMostaque #stablediffusion & @BigscienceW Bloom - aka the coolest new models ;) - have in common?

They both use a new gen of ML licenses aimed at making ML more open & inclusive while keeping it harder to do harm with them. So cool!

https://t.co/o5ELwjc6vk

[BigScience Large Model Training @BigScienceLLM]

@bparlan You can read more about the language selection in this tweet thread: https://t.co/PG2cZPcIyY

[Yacine Jernite @YJernite]

The question "why wasn't language X included in the @BigScienceLLM training data" often comes up

The final list was a consequence of both the project's driving values and of its community-driven nature, here's a quick overview of what happened:

1/7

[Hugging Face @huggingface]

The Technology Behind BLOOM Training🌸

Discover how @BigscienceW used @MSFTResearch DeepSpeed + @nvidia Megatron-LM technologies to train the World's Largest Open Multilingual Language Model (BLOOM):

https://t.co/8QOxhrIVbs

[BigScience Research Workshop @BigscienceW]

BLOOM is here. The largest open-access multilingual language model ever. Read more about it or get it at
https://t.co/mE013I62In
https://t.co/KrBRVklXLf

[Saulnier Lucile @LucileSaulnier]

🌸@BigscienceW BLOOM's intermediate checkpoints have already shown some very cool capabilities!

What's great about BLOOM is that you can ask it to generate the rest of a text - and this even if it is not yet fully trained yet! 👶

🧵 A thread with some examples

[BigScience Large Model Training @BigScienceLLM]

▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 102%

[BigScience Large Model Training @BigScienceLLM]

For 111 days, we've enjoyed world-class hardware stability and throughput thanks to the hard work of our friends at @Genci_fr, @INS2I_CNRS, Megatron & DeepSpeed. Having reached our objective earlier than expected, we'll keep training for a few more days. Stay tuned, more soon ;)

[BigScience Large Model Training @BigScienceLLM]

▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 101%

[BigScience Large Model Training @BigScienceLLM]

▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 100%

[BigScience Large Model Training @BigScienceLLM]

▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 99%

[BigScience Large Model Training @BigScienceLLM]

▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 98%

[BigScience Large Model Training @BigScienceLLM]

▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 97%

[BigScience Large Model Training @BigScienceLLM]

▓▓▓▓▓▓▓▓▓▓▓▓▓▓░ 96%

[BigScience Large Model Training @BigScienceLLM]

▓▓▓▓▓▓▓▓▓▓▓▓▓▓░ 95%

[BigScience Large Model Training @BigScienceLLM]

▓▓▓▓▓▓▓▓▓▓▓▓▓▓░ 94%

[BigScience Large Model Training @BigScienceLLM]

▓▓▓▓▓▓▓▓▓▓▓▓▓▓░ 92%