Royan Dawud Aldian @rdaldian - Twitter Profile

rdaldian retweeted

over 2 years ago

New open weights LLM from @MistralAI params.json: - hidden_dim / dim = 14336/4096 => 3.5X MLP expand - n_heads / n_kv_heads = 32/8 => 4X multiquery - "moe" => mixture of experts 8X top 2 👀 Likely related code: https://t.co/txsnrlriMt Oddly absent: an over-rehearsed professional release video talking about a revolution in AI. If people are wondering why there is so much AI activity right around now, it's because the biggest deep learning conference (NeurIPS) is next week.

karpathy's tweet photo. New open weights LLM from @MistralAI

params.json:
- hidden_dim / dim = 14336/4096 => 3.5X MLP expand
- n_heads / n_kv_heads = 32/8 => 4X multiquery
- "moe" => mixture of experts 8X top 2 👀

Likely related code:
https://t.co/txsnrlriMt

Oddly absent: an over-rehearsed professional release video talking about a revolution in AI.

If people are wondering why there is so much AI activity right around now, it's because the biggest deep learning conference (NeurIPS) is next week.

76

5K

507

1K

797K

rdaldian retweeted

Mistral AI

@MistralAI

over 2 years ago

magnet:?xt=urn:btih:5546272da9065eddeb6fcd7ffddeef5b75be79a7&dn=mixtral-8x7b-32kseqlen&tr=udp%3A%2F%https://t.co/uV4WVdtpwZ%3A6969%2Fannounce&tr=http%3A%2F%https://t.co/g0m9cEUz0T%3A80%2Fannounce RELEASE a6bbd9affe0c2725c1b7410d66833e24

518

10K

2K

4K

5M

rdaldian retweeted

Sophia Yang, Ph.D.

@sophiamyang

over 2 years ago

What is Mixture-of-Experts (MoE)? MoE is a neural network architecture design that integrates layers of experts/models within the Transformer block. As data flows through the MoE layers, each input token is dynamically routed to a subset of the experts for computation. This approach allows for more efficient compute with better results as each expert becomes specialized at particular tasks. Key components: - Experts: MoE layers comprise many experts, small MLPs or complex LLMs like Mistral 7B. - Router: Routers determine which input tokens get assigned to which experts. There are two routing strategies: token chooses the router or router chooses the token. How does it work exactly? It uses a softmax gating function to model a probability distribution through experts or tokens and choose the top k. Why MoE? - Each expert can be specialized to handle different tasks or different parts of the data. - Adds learnable parameters to LLMs without increasing inference cost - Can utilize efficient computation on sparse matrices - Computes all expert layers in parallel to effectively use the parallel capabilities of GPUs - Helps scale the model efficiently with reduced training time. Better results at lower computing costs! Papers to read: - The Sparsely-Gated Mixture-of-Experts Layer (2017): https://t.co/laOv9xoLvJ - GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020) https://t.co/fUhqA8CVIH - MegaBlocks: Efficient Sparse Training with Mixture-of-Experts (2022): https://t.co/POrZJW9BtO - Mixture-of-Experts Meets Instruction Tuning (2023): https://t.co/fcnbNSO0HV

sophiamyang's tweet photo. What is Mixture-of-Experts (MoE)?

MoE is a neural network architecture design that integrates layers of experts/models within the Transformer block. As data flows through the MoE layers, each input token is dynamically routed to a subset of the experts for computation. This approach allows for more efficient compute with better results as each expert becomes specialized at particular tasks.

Key components:

- Experts: MoE layers comprise many experts, small MLPs or complex LLMs like Mistral 7B.
- Router: Routers determine which input tokens get assigned to which experts. There are two routing strategies: token chooses the router or router chooses the token. How does it work exactly? It uses a softmax gating function to model a probability distribution through experts or tokens and choose the top k.

Why MoE?

- Each expert can be specialized to handle different tasks or different parts of the data.
- Adds learnable parameters to LLMs without increasing inference cost
- Can utilize efficient computation on sparse matrices
- Computes all expert layers in parallel to effectively use the parallel capabilities of GPUs
- Helps scale the model efficiently with reduced training time. Better results at lower computing costs!

Papers to read:

- The Sparsely-Gated Mixture-of-Experts Layer (2017): https://t.co/laOv9xoLvJ
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020) https://t.co/fUhqA8CVIH
- MegaBlocks: Efficient Sparse Training with Mixture-of-Experts (2022): https://t.co/POrZJW9BtO
- Mixture-of-Experts Meets Instruction Tuning (2023): https://t.co/fcnbNSO0HV

26

2K

380

2K

335K

rdaldian retweeted

PyTorch

@PyTorch

about 6 years ago

Here's the list of live PyTorch Q&As on Wednesday, July 15 at #icml2020. The conference attendees can find links in the Additional Resources section within the virtual booth: https://t.co/GT8nOCZhlD

PyTorch's tweet photo. Here's the list of live PyTorch Q&As on Wednesday, July 15 at #icml2020. The conference attendees can find links in the Additional Resources section within the virtual booth: https://t.co/GT8nOCZhlD https://t.co/ONWFUSnlg3

1

265

57

25

0

Who to follow

You'll never walk alone

Royan Dawud Aldian

@rdaldian

Who to follow

Last Seen Users on Sotwe

Trends for you

Most Popular Users