Truss now supports automatic caching for weights from Hugging Face ๐ค
Just add `hf_cache` to your `config.yml` with a valid `repo_id`.
It's a simple way to reduce cold start times!
But what does that mean? Let's unpack it ๐
(5/6)
`hf_cache` has a few subkeys:
- `revision`: your Hugging Face revision (defaults to `main`)
- `allow_patterns`: cache only files that match certain patterns.
- `ignore_patterns`: similarly, define files to ignore.
Patterns specified must follow Unix shell-style wildcards.
Generating LLM outputs can take a while! This can lead your users to wonder if something went wrong, or even bounce from your app. Now, with streaming available in Truss, your users can start getting results immediately!
Deploy all the versions of Llama-2 right now with @baseten!
Base
7B - https://t.co/j7lZzdG3cx
13B - https://t.co/MXzLzIEEKt
70B - https://t.co/gOMZx07lEX
Chat
7B - https://t.co/08Z880Esor
13B - https://t.co/jaG4X8ihF2
70B - https://t.co/iVp1hKnZV5