Alright!
I was able to achieve decent facial consistency with the use of makeup to outline the basic shapes and details I was looking for.
#stablediffusion
I trained this @ltx_model LTX 2.3 LoRA of George Costanza at home on my 5090 in about a day with AI Toolkit. I generated this 30 second video with @ComfyUI on my 5090 in 6 minutes. Open source is, always has been, and always will be, the future of generative AI. (SOUND ON)
YO,
I adapted VACE to work with real-time autoregressive video generation.
Here's what it can do right now in real time:
- Depth, pose, optical flow, scribble, edge maps โ all the v2v control stuff
- First frame animation / last frame lead-in / keyframe interpolation
- Inpainting with static or dynamic masks
- Stacking stuff together (e.g. depth + LoRA, inpainting + reference images)
- Reference-to-video is in there too but quality isn't great yet compared to batch
Getting ~20 fps for most control modes on a 5090 at 368x640 with the 1.3B models. Image-to-video hits ~28 fps. Works with 14b models as well, but doesnt fit on 5090 with VACE.
This is all part of Daydream Scope, an open source tool for running real-time interactive video generation pipelines. The demos were created in/with Scope, and is a combination of Longlive, VACE, and Custom LoRA.
There's also a very early WIP ComfyUI node pack wrapping Scope.
But how is a real-time, autoregressive model relevant to @ComfyUI ? Ultra long video generation. You can use these models distilled from Wan to do V2V tasks on thousands of frames at once, technically infinite length. I havent experimented much more than validating the concept on a couple thousand frames gen. It works!
Full technical details on real-time VACE + more examples here (link in comments)
Curious what people think. Happy to answer questions.
Video + Custom LoRA links also in comments.
Love,
Ryan
p.s. I will be back with a sick update on ACEStep implementation tomorrow
(links in first comment)
โฐ Ace-Step Training for comfyUI
you can train your own loRa's in comfy with my custom node pack!
This is still in development so any PR's or bug reports via Github are appreciated
Link below๐
We're releasing ACE-Step-v1.5(2B), a fast, high-quality open-source music model.
It runs locally on a consumer-grade GPU, generates a full song in under 2 seconds(on an A100), supports LoRA fine-tuning, and beats SUNO on common eval metrics.
GitHub: https://t.co/Zydfhw9oUl
Key traits:
Quality: beats Suno on common eval scores
Speed: full song under 2s on A100
Local: ~4GB VRAM, under 10s on RTX 3090
LoRA: train your own style with a few songs
License: MIT, free for commercial use
Data: fully authorized plus synthetic
The music AI space lacks commercial-grade open models. Many creators are forced to rely on closed-source services, and canโt fully own, run locally, or fine-tune their own models. We want to help change that.
Got Training working inside comfy for AceStep 1.5!
Doing a longer training run now to triple check my implementation is correct. If i get the green light i'll be releasing the node pack soon!
Stay Tuned.
โฐIm excited to drop my LTX-2 i2v adapter LoRa open Source!
This LoRa was trained on 30,000 videos in hopes to enhance LTX-2 Into a much more robust image to video model.
Full scope on the hugging face!
https://t.co/sppLtlOnJ0
LTX-2 Image-to-Video Adapter LoRA by @Machinedelusion! Massive work!
- boosts motion and quality for i2v;
- Trained on 30K videos.
Itโs not a style-specific fine-tune, but a general-purpose enhancer.
https://t.co/DeqHuKVPN2
@ComfyUI I find it pretty annoying that the Founderโs Edition doesnโt include model importing.
Do I want bonus credits or full functionality?
Feels like Iโm being forced to give up the bonus just to get full features.
A new open source wave is about to hit, and @ostrisai is ushering it in, in full force. Loraโs with audio opens up an insane amount of possibilities for creative exploration and utility
Q1 is exciting already!