Lingyun (Stephen) Yang

Senior Engineer @AlibabaGroup | CS Ph.D. @HKUST | AI Infra Researcher | Making AI accessible to everyone

[Lingyun (Stephen) Yang @stephenyang1999]

@JeffDean Good luck to the next chapter!

[Lingyun (Stephen) Yang @stephenyang1999]

📜 Want to learn more about Katz? Read our full #ATC25 paper:
https://t.co/ryEksa7XU8

🌟 Katz is open source! Get the code and try it out:
https://t.co/nrW5UFeKaC

4/4

[Lingyun (Stephen) Yang @stephenyang1999]

🎉 Excited to present Katz, our high-performance serving system for diffusion models with multiple adapters, at #ATC25!

#DiffusionModels #ModelServing

1/n

[Lingyun (Stephen) Yang @stephenyang1999]

📊 Katz delivers impressive performance gains for Stable Diffusion XL workflows on NVIDIA H800 GPUs:

⚡️ Up to 7.8× lower end-to-end latency
💪 Up to 1.7× higher inference throughput

🖼️ All while maintaining great image quality!

3/n

[Lingyun (Stephen) Yang @stephenyang1999]

I’m excited to be at ATC 2025 & OSDI 2025! If you're interested in diffusion model serving or GPU cluster scheduling, feel free to reach out! Find me at the conference venue and let's chat about the latest research. #atc25 #osdi25 #SystemsResearch #MLSys

[Ryan Huang @ryanphuang]

We are seeking members to join the Artifact Evaluation Committee for #OSDI25 & #USENIXATC25.

Help promote reproducibility, and engage with cutting-edge systems research!

Please apply by April 17th via https://t.co/OqY9nFLhmI.

#SystemsResearch #Reproducibility #OSDI #ATC

[Lingyun (Stephen) Yang @stephenyang1999]

@xiangfeng_zhu Congrats!

[Lingyun (Stephen) Yang @stephenyang1999]

@yqsong Congrats!

[Francis Y. Yan @FrancisYan_]

Grace Liu, @yangzhouy, and I are co-chairing the SIGCOMM 2024 Artifact Evaluation. We are looking for PhD students, postdocs, and early-career researchers to join the committee! Underrepresented groups are strongly encouraged to apply.
- Application deadline: June 12, 2024 (AoE)
- Self-nomination form: https://t.co/dFFLZsj6DL

[ICLR @iclr_conf]

Announcing the #ICLR2024 Outstanding Paper Awards:
https://t.co/0k9LJwNE6o

Shoutout to the awards committee: @eunsolc, @katjahofmann, @liu_mingyu, @nanjiang_cs, @guennemann, @optiML, @tkipf, @CevherLIONS

[Yeming Wen @YemingW]

🚀 Excited to share that our paper "Batched Low-Rank Adaptation of Foundation Models (https://t.co/xmqygWBa6V)" was accepted for an oral presentation at #ICLR2024! We're advancing parameter efficient fine-tuning (PEFT) in LLMs for more personalized and efficient AI. (1/n)

[Shubham Saboo @Saboo_Shubham_]

Mistral AI releases first ever OpenSource MoE model

Fun facts:
- Released as 87 GB torrent
- Seems to be a scaled down version of GPT-4
- Released on X, no press release and refuse to elaborate

Google left the AI community in awe with the over rehearsed demo video that's now getting overly criticised.

While the Opensource AI startup Mistral AI released a MoE model consisting of 8 7B experts.

What's Mixture of Experts (MoE)?

Mixture of Experts (MoE) is a technique used in LLMs to enhance their efficiency and accuracy. This approach works by dividing complex tasks into smaller, more manageable sub-tasks, each handled by a specialized mini-model or "expert".

Here's a concise breakdown:

1. Expert Layers: These are smaller neural networks trained to be highly skilled in specific areas. Each expert processes the same input but in a way that aligns with its unique specialization.

2. Gating Network: This is the decision-maker of the MoE architecture. It evaluates which expert is best suited for a given piece of input data. The network calculates a compatibility score between the input and each expert, and then uses these scores to determine the involvement of each expert in the task.

Together, these components ensure that the right expert handles the right task. The gating network effectively routes each input to the most appropriate expert(s), while the experts focus on their specific areas of strength. This collaborative training leads to a more versatile and capable overall model.

Details around Mistral's new MoE (from Reddit)

For the inference of each token, only 2 experts are used. This information can be derived from the model metadata:

{"dim": 4096, "n_layers": 32, "head_dim": 128, "hidden_dim": 14336, "n_heads": 32, "n_kv_heads": 8, "norm_eps": 1e-05, "vocab_size": 32000, "moe": {"num_experts_per_tok": 2, "num_experts": 8}

Comparison with GPT-4

Mistral 8x7B uses a very similar architecture to that of GPT-4, but scaled down:

- 8 total experts instead of 16 (2x reduction)
- 7B parameters per expert instead of 166B (24x reduction)
- 42B total parameters (estimated) instead of 1.8T (42x reduction)
- Same 32K context as the original GPT-4

[Neeraja Yadwadkar @NeerajaJY]

As I will soon start looking at the grad applications, here is a reminder to send me a quick email if you have applied and would like to work with us! We are a close-knit group of UT-SysML at UT Austin ECE 🤗!

[Ilya Sutskever @ilyasut]

I deeply regret my participation in the board's actions. I never intended to harm OpenAI. I love everything we've built together and I will do everything I can to reunite the company.

[Satya Nadella @satyanadella]

We remain committed to our partnership with OpenAI and have confidence in our product roadmap, our ability to continue to innovate with everything we announced at Microsoft Ignite, and in continuing to support our customers and partners. We look forward to getting to know Emmett Shear and OAI's new leadership team and working with them. And we’re extremely excited to share the news that Sam Altman and Greg Brockman, together with colleagues, will be joining Microsoft to lead a new advanced AI research team. We look forward to moving quickly to provide them with the resources needed for their success.

[OpenAI @OpenAI]

OpenAI announces leadership transition https://t.co/fFYDLwGXQz

[HKUST Computer Science and Engineering @HKUSTCSE]

HKUST CSE is recruiting!!!
See how to apply👇
https://t.co/rOfWaVfNrc

[Jonathan Aldrich @JAldrichPL]

FAQ: Why use a publisher vs. put stuff on the web?
(1) Stability. You want an org you can count on to make your papers available in 100 years. Publishers have contractual backups: if ACM goes under, another org will step in with a commitment to make the papers available forever.

[Jonathan Aldrich @JAldrichPL]

Surprising fact: no major professional scientific organization has flipped from a paywalled publishing model to 100% Open Access.

@TheOfficialACM is going to be the first on January 1, 2026, when its entire digital library will become freely available to the public.