Patrick Galbraith

Software Engineer | Heavy Metal🤘fan | Writing #fantasy and experimenting with #ai | Obsessive creator of software, novels, art, web things... (He/Him)

[Patrick Galbraith @P_Galbraith]

My daughter loves filling in the black areas in my sketchbook. Lots of happy accidents 😅

[Patrick Galbraith @P_Galbraith]

My brother's latest masterpiece! - Avalon Rising: The Tabletop Role-Playing Game on @Kickstarter https://t.co/LG7YJFpUna

[Patrick Galbraith @P_Galbraith]

100fps Stable Diffusion! Seems like>30fps img2img too which is pretty amazing.

[AK @_akhaliq]

StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation

paper page: https://t.co/LF4vb6jV3o

introduce StreamDiffusion, a real-time diffusion pipeline designed for interactive image generation. Existing diffusion models are adept at creating images from text or image prompts, yet they often fall short in real-time interaction. This limitation becomes particularly evident in scenarios involving continuous input, such as Metaverse, live video streaming, and broadcasting, where high throughput is imperative. To address this, we present a novel approach that transforms the original sequential denoising into the batching denoising process. Stream Batch eliminates the conventional wait-and-interact approach and enables fluid and high throughput streams. To handle the frequency disparity between data input and model throughput, we design a novel input-output queue for parallelizing the streaming process. Moreover, the existing diffusion pipeline uses classifier-free guidance(CFG), which requires additional U-Net computation. To mitigate the redundant computations, we propose a novel residual classifier-free guidance (RCFG) algorithm that reduces the number of negative conditional denoising steps to only one or even zero. Besides, we introduce a stochastic similarity filter(SSF) to optimize power consumption. Our Stream Batch achieves around 1.5x speedup compared to the sequential denoising method at different denoising levels. The proposed RCFG leads to speeds up to 2.05x higher than the conventional CFG. Combining the proposed strategies and existing mature acceleration tools makes the image-to-image generation achieve up-to 91.07fps on one RTX4090, improving the throughputs of AutoPipline developed by Diffusers over 59.56x. Furthermore, our proposed StreamDiffusion also significantly reduces the energy consumption by 2.39x on one RTX3060 and 1.99x on one RTX4090, respectively.

[Patrick Galbraith @P_Galbraith]

@aipixelfusion @thibaudz It works great for colouring lineart but requires an input image. https://t.co/o7oLmIynZy

[Patrick Galbraith @P_Galbraith]

Youtuber @shadmbrooks posted this AI challenge and a lot of people have been struggling to recreate this image. Anyone who has used AI before knows that this isn't possible with txt2img but there is a way to do it in a single shot using #ControlNet 🧵

[Patrick Galbraith @P_Galbraith]

@emmanuel_2m @hervenivon Yeah using LoRA + Scribble ControlNet works pretty well. Another alternative which gives one-shot results is using IP-Adapter which avoids the need for LoRA training, especially if you have tighter lineart. https://t.co/o7oLmIynZy

[Patrick Galbraith @P_Galbraith]

Youtuber @shadmbrooks posted this AI challenge and a lot of people have been struggling to recreate this image. Anyone who has used AI before knows that this isn't possible with txt2img but there is a way to do it in a single shot using #ControlNet 🧵

[Patrick Galbraith @P_Galbraith]

@WebCoderz Just --xformers

[Patrick Galbraith @P_Galbraith]

Tutorial for realtime prompting in #automatic1111 using SDXL Turbo! 0.3s per image🔥

[Brendan Bycroft @BrendanBycroft]

Project #2: LLM Visualization

So I created a web-page to visualize a small LLM, of the sort that's behind ChatGPT. Rendered in 3D, it shows all the steps to run a single token inference. (link in bio)

[Patrick Galbraith @P_Galbraith]

@jamonete782 May be GPU? I'm using RTX3090.

[Patrick Galbraith @P_Galbraith]

@bcjordan RTX3090, some people are saying 4090 is around 100ms

[Patrick Galbraith @P_Galbraith]

@the_gonchar @_akhaliq You can use high res fix but it's a bit inconsistent. There are some 1024px turbo models on CivitAI but they're a bit slower and need higher step counts.

[Patrick Galbraith @P_Galbraith]

@furkangozukara @_akhaliq Yeah it is unfortunately

[Patrick Galbraith @P_Galbraith]

How far things have come in just over 12 months⚡️

I did the piece on the right using IMG2IMG from my sketch just after the release of #StableDiffusion last year. Which at the time was better than Dalle for artistic control. But look at the updated version using #ControlNet!

[Patrick Galbraith @P_Galbraith]

@thegoldengod94 @AIandDesign @Magnific_AI Not sure what you mean by jailbroken (uncensored...?). Checkout https://t.co/OoJG0NIIwD there are lots of free models around. You will have trouble running the big ones so stuck to the 9B/13B ones to start with.

[Patrick Galbraith @P_Galbraith]

@thegoldengod94 @AIandDesign @Magnific_AI Just use compfyui or automatic1111. There are better upscaling workflows when running locally if you have the know-how.

[Patrick Galbraith @P_Galbraith]

@mankrip @chaseleantj Would be easy to do using stable diffusion img2img at low strength and ControlNet, just need to prompt it correctly.

[Patrick Galbraith @P_Galbraith]

Experimenting with a new aesthetic

[Patrick Galbraith @P_Galbraith]

@jimclydego @dreamwieber Probably just Stable Diffusion + LCM-Lora + ControlNet

[Patrick Galbraith @P_Galbraith]

Here is the change I made to the automatic1111 source https://t.co/XCQDnGjsAM

Right click "generate" and select "Generate forever"

[Patrick Galbraith @P_Galbraith]

Real time AI painting with #automatic1111 and Photoshop! Managed to modify #automatic1111 to support Img2Img from webcam stream. Combined with the speed of LCM-Lora and the "generate forever" option gives almost real time feedback. Which is super fun!

[Patrick Galbraith @P_Galbraith]

Incredibly impressed by Mistral 7b. Wouldn't be surprised to see this replace GPT3.5 for some workloads with fine-tuning. Crazy to see how far local LLM have come.