mosec v0.8.0 is out! 🎉https://t.co/FOLZ4oaKCs
New features:
- multi-model serving with shared workers
- OpenAPI generation
- SSE support
There is a model serving benchmark that compares mosec with pytriton: https://t.co/oJYLRaorfX
We have added the Performance Tuning Guide in README to help you get the most out of your computing resources. 😋
BTW, we will release some benchmarks later. Stay tuned.
https://t.co/kIKxKFXcl8
v0.6.7 has been released! 🥳https://t.co/7BqsBVK3qL
Now we support TypedMsgPackMixin for request validation with the help of #msgspec. Thanks @jcristharif for the awesome library. Check the doc here: https://t.co/QHraHEUZYr
We have released v0.6.5 🥳 https://t.co/5O6ILlmKWe
Now you can specify a timeout for the worker forward process. So the worker won't be blocked too long by some mistakes in the code.
https://t.co/wuYQKkpEYE
MOSS is a fascinating chatbot from @tianxiangsun@FudanUniv in its early stages of development. At the heart of this technology lies our open-sourced project https://t.co/fJsVA1Ihe6
Why not explore them if you're eager to discover more about LLM?
We have a #stablediffusion serving example!
You can try this one on your laptop. GTX 3070 should be enough to run this demo.
BTW, we provide an #envd file so you can improve this demo in the container development env. For the detail, check @tensorchord.
https://t.co/VBQ5Q3hbQ3