Today I made a prototype of a system that turns a single AI-generated portrait into a living VTuber. No rigging, no Live2D, nobody spending 80 hours in an editor.
The character was generated locally with SDXL using the same model and custom LoRA I use for the art bot in my Discord server.
A video model (Grok) then animated her into a chain of idle clips, each one starting on the exact frame where the previous one ended, so she loops forever and rarely cuts.
Her mouth is the interesting part: a tracker follows the drawn lines of her lips through every frame, measuring position, angle, and width, and a composited mouth rides along, rotating and foreshortening as her head moves. The mouth shapes themselves were inpainted onto her own face by SDXL, so every viseme is in her exact art style. Real phoneme detection drives which shape shows.
Because the mouth is composited rather than baked in, it stops the instant the voice stops. The body animation never even knows speech exists.
Animating her ate about half of my SuperGrok credits, which works out to $14.32 if you paid for the same experimentation directly. I used 5 of the 12 generated clips in this demo. Narrated this demo using the Kokoro TTS model for free.
Next step would be webcam emotion detection so she can swap between animation sets when the streamer smiles or gasps, more mouth shapes
Maybe I’ll never get the point of AI art, just because I enjoy the drawing process too much.
"AI will replace you" aaah, cool, i guess???... I still love my hobby anyway ( ̄ヘ ̄)
【 GIVEAWAY TIME! 】End: July 24th
Wanna get FLOOFED? I'm giving away 5 #FLOOFcustomizable models, PLUS one extra for every 100 sales!
TO ENTER:
‣ 🩷 + RT + Follow!
‣ Extra entries: drop your PNG + RT the original post below!
This weekend, our safety systems incorrectly triggered and banned around 200 accounts. Everyone affected has been reinstated.
Here's what happened. A thread 🧵