Over the past few weeks, I've been working on optimizing neural text-to-speech inference using adaptive compute and caching with dots.tts at @uwwatai.
Last week, our team, Adaptive-Compute Inference for Real-Time TTS, led by @saikodakalla19, gave a talk @UWaterloo about some of our findings.
One of our biggest findings: 98.5% of traced repeated-operation time was spent inside the audio refinement model.
Here's what I worked on and what we discovered! ↓