Michele Zanotti

Software Engineer @ Mistral AI

[Arthur Mensch @arthurmensch]

It's actually le gros chaton

[Diego @diegofiori_]

@learnk8s Congrats @Telemaco019 for the project!

[Kubernetes Community Days Italy @KCDItaly]

.@Telemaco019 will explore how to increase GPU utilization through GPU sharing technologies, including Time Slicing, MIG, and MPS, and discuss how they can be used in Kubernetes. We will then focus on how to achieve effective GPU sharing through Dynamic GPU Partitioning 👏🏻👏🏻👏🏻

[Michele Zanotti @Telemaco019]

I'm thrilled to announce that I'll be speaking at #kubernetes Community Days Italy 2023! 😍 I'm looking forward to it, see you in Milan next June 16th!

[Kubernetes Community Days Italy @KCDItaly]

More speakers to introduce today!
We take this opportunity to thank those people who submitted and weren't chosen: you are great, keep submitting!

Now, you all, grab the tickets to attend the talks of these fantastic folks! 🤓🥳

@KubernetesDays #kcditaly

[Michele Zanotti @Telemaco019]

If you want to remain updated on the latest Nebuly news join our Discord community https://t.co/lwJnHnu33B

3/3

[Michele Zanotti @Telemaco019]

Just released! #nos 0.1.1 🚀

https://t.co/ya8w5bu12x

🔨 Fix Dynamic MIG Partitioning on multi-GPU nodes
✨ The GPUs of MIG-enabled nodes are now initialized with the largest available MIG profile
✨ Add new GPU models to known MIG geometries
✨ Improve documentation

1/

[Michele Zanotti @Telemaco019]

Special thanks to all the community members that helped us troubleshoot issues and provided extremely valuable feedback on how to improve #nos!

2/

[Michele Zanotti @Telemaco019]

I will dig into the results and write a short article in the next few days. In the meantime, here you can find the link to the source code used to run the experiments: https://t.co/WkHmVQvX9I

9/9

#Kubernetes #NVIDIA #GPU #MLOps

[Michele Zanotti @Telemaco019]

NVIDIA Multi-Process Service (MPS) is not a very popular GPU sharing technology, but it has great potential.

One of the main advantages of MPS over Time-Slicing is that processes are run in parallel instead of concurrently, so there is no context-switching overhead.

2/

[Michele Zanotti @Telemaco019]

➡️ It’s also interesting to note that when sharing the GPU between 7 Pods, MPS gives slightly better performance than MIG.

8/

[Michele Zanotti @Telemaco019]

Is MPS better than MIG and Time-Slicing for performing inference on a shared GPU? My initial tests show better performance both when I shared the GPU between 1 and many inference Pods.

1/

[Michele Zanotti @Telemaco019]

@AbdElRahmanAMA Hi @AbdElRahmanAMA, sure! What's the matter? We can discuss it on Nos Discord channel https://t.co/9whuebL2ZN

[Michele Zanotti @Telemaco019]

New article on Towards Data Science! In this article I compare existing GPU sharing techniques on Kubernetes and delve into MPS and how to use it to dynamically share GPUs.

https://t.co/XTLbqDjQfn

1/

[Michele Zanotti @Telemaco019]

Main cons

👷 Even though MPS allows setting memory limits on each process, it does not provide error isolation and memory protection (unlike MIG).

What do you think about MPS?

#kubernetes #mlops #devops #ai #nvidia #gpu #infrastructure

3/3

[Michele Zanotti @Telemaco019]

MPS is not a well-known, but very powerful GPU sharing technology.

Pros

⚡ Processes run in parallel (not concurrently, as with Time Slicing)

🤖 Supported by almost all GPU architectures

🪴 Fine-grained control of memory allocation and compute resources.

2/