Ziji Shi

Ph.D. at @NUScomputing. Prev. @Google XLA. @Apple @alibaba_cloud & @sensetime_ai. I build highly efficient systems for ML.

[Ziji Shi @Shi23Steven]

@cHHillee Perhaps privacy? Industry-scale RL post-train needs to interact with the internal environment, dashboards, and user data.

[DeepSpeed @DeepSpeedAI]

Introducing Domino: a novel zero-cost communication tensor parallelism (TP) training engine for both single node and multi-node settings.

- Near-complete communication hiding
- Novel multi-node scalable TP solution

Blog: https://t.co/08bPanyr9M

[HPC Papers @HPCPapers]

ParaGAN: A Scalable Distributed Training Framework for Generative
Adversarial Networks by Ziji Shi et al.
https://t.co/n3pQJEAzad

[Ziji Shi @Shi23Steven]

@davmoltisanti @UniofBath Congrats Davide!

[Fuzhao Xue (Frio) @XueFz]

1/ Announcing the development of OpenMoE project! 🚀

Open Mixture-of-Experts Language Models!
MoE + UL2 objective + umT5 tokenizer + 50% code data mix.

GitHub: https://t.co/dxcMsFRKY3
Blog: https://t.co/AznT0PfdFE

[Ziji Shi @Shi23Steven]

I’m attending #ISCA co-located with #FCRC 🎉 We will present two papers at the MlArchSys and ASSYST workshop on #LLM and #GAN at Canary 2. Feel free to drop by and say hi! @ISCAConfOrg

[Ziji Shi @Shi23Steven]

(2/2) We warmly welcome all professionals in the field to join us, engage in enriching conversations, and contribute to our vibrant community! #LLM #Singapore #MLSys #AI #CommunityBuilding 🤝

[Ziji Shi @Shi23Steven]

(1/2) As large-scale models continue to evolve, the need for associated foundational systems is also growing. We've set up an MLSys discussion group (https://t.co/R5qcTlglXM), planning to host bi-weekly discussions on academic papers or updates on cutting-edge advancements.

[Ziji Shi @Shi23Steven]

(1/2) As large-scale models continue to evolve, the need for associated foundational systems is also growing. We've set up an MLSys discussion group (https://t.co/R5qcTlglXM), planning to host bi-weekly discussions on academic papers or updates on cutting-edge advancements.

[Zijiao Chen @ZijiaoC]

🧵🧠 We're witnessing incredible scientific progress in image & text reconstruction from fMRI nowadays. But what about reconstructing video from fMRI? Allow me to introduce our recent preprint: Mind-Video
https://t.co/VL2KXz8o9K
https://t.co/KyNtsCxDIJ
https://t.co/bhjz0PDlS6

[Ziji Shi @Shi23Steven]

@XueFz @Francis_YAO_ From the HW point of view, I think sparse computation ( eg. cuSPARSE) can be used to accelerate the sparse activations, while the MoE models may not, as they operate on a much coarser level. But some architectural changes may help (eg load balancing among experts etc).

[Ziji Shi @Shi23Steven]

@XueFz @Francis_YAO_ What do you mean by *sparsely activated* here for dense models?

[Ziji Shi @Shi23Steven]

@YiTayML Happy Birthday! All the best with the startup!

[Ziji Shi @Shi23Steven]

@YisongMiao Thanks! The DAG representation of NN was originally proposed by others (eg. #TensorFlow ), but we leverage it to shrink the search space for parallel strategy into a much smaller domain, where we are more comfortable optimizing. Hopefully, it paves the way for #LLM research :)

[cs.LG Papers @arxiv_cs_LG]

TAP: Accelerating Large-Scale DNN Training Through Tensor Automatic Parallelisation. Ziji Shi, Le Jiang, Ang Wang, Jie Zhang, Xianyan Jia, Yong Li, Chencan Wu, Jialin Li, and Wei Lin https://t.co/d4BnMvXMBT

[Ziji Shi @Shi23Steven]

We recently uploaded our work with Alibaba on *quickly* and *automatically* finding the optimal #tensorparallel strategy for #LLM. Compared to SoTA approaches, we are ~20-160x faster. Comments are welcomed!

Arxiv: https://t.co/mxEhMrKF8R

[Ziji Shi @Shi23Steven]

#ChatGPT has been phenomenal, but have you ever wondered how it was trained? In fact, finding the optimal parallel strategy for such LLM is very challenging, as the candidate space grows exponentially w.r.t size. (1/2)

[Ziji Shi @Shi23Steven]

#ChatGPT has been phenomenal, but have you ever wondered how it was trained? In fact, finding the optimal parallel strategy for such LLM is very challenging, as the candidate space grows exponentially w.r.t size. (1/2)

[Ziji Shi @Shi23Steven]

@Gaoshen39568652 Thanks mate!

[Ziji Shi @Shi23Steven]

Eagerly expecting a break after the ASPLOS deadline...

[Ziji Shi @Shi23Steven]

@changsheng_sun @TheOfficialACM Thanks!

[Ziji Shi @Shi23Steven]

Just received my travel grant for the HPC summer school from @TheOfficialACM. Much appreciated! The notes can be found here: https://t.co/rcOKiBwK0N

[Ziji Shi @Shi23Steven]

@YisongMiao @TheOfficialACM 😁