mosec

Machine Learning Model Serving made Efficient in the Cloud.

[mosec @mosecorg]

mosec has adopted Ruff @astral_sh
https://t.co/zvc83ZOJhV

[mosec @mosecorg]

mosec v0.8.0 is out! 🎉https://t.co/FOLZ4oaKCs
New features:
- multi-model serving with shared workers
- OpenAPI generation
- SSE support
There is a model serving benchmark that compares mosec with pytriton: https://t.co/oJYLRaorfX

[mosec @mosecorg]

We have added the Performance Tuning Guide in README to help you get the most out of your computing resources. 😋
BTW, we will release some benchmarks later. Stay tuned.
https://t.co/kIKxKFXcl8

[mosec @mosecorg]

@jcristharif We are glad to be one of them! Thanks 😋

[mosec @mosecorg]

v0.6.7 has been released! 🥳https://t.co/7BqsBVK3qL
Now we support TypedMsgPackMixin for request validation with the help of #msgspec. Thanks @jcristharif for the awesome library. Check the doc here: https://t.co/QHraHEUZYr

[mosec @mosecorg]

We have released v0.6.5 🥳 https://t.co/5O6ILlmKWe
Now you can specify a timeout for the worker forward process. So the worker won't be blocked too long by some mistakes in the code.

[TensorChord @TensorChord]

https://t.co/wuYQKkpEYE

MOSS is a fascinating chatbot from @tianxiangsun @FudanUniv in its early stages of development. At the heart of this technology lies our open-sourced project https://t.co/fJsVA1Ihe6

Why not explore them if you're eager to discover more about LLM?

[mosec @mosecorg]

We have a #stablediffusion serving example!
You can try this one on your laptop. GTX 3070 should be enough to run this demo.
BTW, we provide an #envd file so you can improve this demo in the container development env. For the detail, check @tensorchord.
https://t.co/VBQ5Q3hbQ3

[mosec @mosecorg]

We have released v0.4.1!
Now we support msgpack, no more JSON + base64 for image data. It's also faster and lighter than JSON.
https://t.co/eviGQvS4MN

[mosec @mosecorg]

We have released v0.4.0!
Next, we are going to improve the README/documentation part to make it easier to use.
https://t.co/OCJKqBnfRo

[Yuan (Terry) Tang @TerryTangYuan]

Mosec is a high-performance and flexible model serving framework for building ML model-enabled backend and microservices.

🔨 Built by @urcyanide and other @mosecorg contributors with #rustlang and #python.

#MachineLearning #DataScience #microservices

https://t.co/1xVkmmEE1h