Man M. Ho

Research Fellow at UofU | CSE PhD | Interested in #computervision #photography #deeplearning #pathology

[Man M. Ho @minhman_ho]

Deep Photo Scan: Huge update since the last version.
🌟Better restoration/generalization performance.
🌟More reliable evaluation.
🌟More interesting experiments.
Page: https://t.co/ZUkmEMts6m
Pdf: https://t.co/drtCnoIHfE
Abs: https://t.co/jVQCGZFqHZ
👇

[AK @_akhaliq]

Deep Photo Scan: Semi-supervised learning for dealing with the real-world degradation in smartphone photo scanning
pdf: https://t.co/Dl9e0WmWK2
abs: https://t.co/6mxRcGj1Yv

[Man M. Ho @minhman_ho]

GigaPath: Foundation Model for Digital Pathology
https://t.co/pNCmPhwDcC

[AK @_akhaliq]

I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models

paper page: https://t.co/zj3ip5Y8My

Video synthesis has recently made remarkable strides benefiting from the rapid development of diffusion models. However, it still encounters challenges in terms of semantic accuracy, clarity and spatio-temporal continuity. They primarily arise from the scarcity of well-aligned text-video data and the complex inherent structure of videos, making it difficult for the model to simultaneously ensure semantic and qualitative excellence. In this report, we propose a cascaded I2VGen-XL approach that enhances model performance by decoupling these two factors and ensures the alignment of the input data by utilizing static images as a form of crucial guidance. I2VGen-XL consists of two stages: i) the base stage guarantees coherent semantics and preserves content from input images by using two hierarchical encoders, and ii) the refinement stage enhances the video's details by incorporating an additional brief text and improves the resolution to 1280times720. To improve the diversity, we collect around 35 million single-shot text-video pairs and 6 billion text-image pairs to optimize the model. By this means, I2VGen-XL can simultaneously enhance the semantic accuracy, continuity of details and clarity of generated videos. Through extensive experiments, we have investigated the underlying principles of I2VGen-XL and compared it with current top methods, which can demonstrate its effectiveness on diverse data.

[Ziwei Liu @liuziwei7]

🚀🚀Fancy generating high-res (2K~4K) images using Stable Diffusion without an additional super-resolution module?

* Now combining #FreeU with #ScaleCrafter, you can generate 4K images using SDXL for free!

- FreeU: https://t.co/otGwqbhMBb
- ScaleCrafter: https://t.co/yw0tDC8Vfw

[Ziwei Liu @liuziwei7]

🔥🔥We propose #SEINE, a video diffusion model that focuses on generative transition and prediction.

#SEINE supports *video transition generation* and *image-to-video animation*

- Project: https://t.co/1R9hkaXcie
- Paper: https://t.co/Dn1oYYFFpo
- Code: https://t.co/IryTCXbiiJ

[Ziwei Liu @liuziwei7]

🚀🚀We propose #FreeNoise, another *free lunch* that can generate longer videos given multiple prompts, with negligible time cost overhead.

- Project: https://t.co/dZ4QvjnR8s
- Paper: https://t.co/9LlTjZz3UC
- Code: https://t.co/nomd1xzCox

[Ziwei Liu @liuziwei7]

🔥FreeU now has a major upgrade🔥

* By adding structure-aware scaling, FreeU excels at both structural coherence and detail preservation, largely improving the aesthetic quality upon Stable Diffusion XL for free.

- Paper: https://t.co/znpFpYE16x
- Code: https://t.co/otGwqbhMBb

[Man M. Ho @minhman_ho]

Better than LPIPS ! 😲 😲 😲
DreamSim: Learning New Dimensions of
Human Visual Similarity using Synthetic Data

Code: https://t.co/Z2NC5fhAp0

[Ziming Liu @ZimingLiu11]

Mechanistic interpretability is not only for ML or LLM, but is even more promising for Science! A few months ago, we proposed a brain-inspired method (BIMT) for NN interpretability; Now we're happy to see that it can give something back to neuroscience - Growing brains in RNNs!

[Ziwei Liu @liuziwei7]

🔥🔥 We propose #DreamGaussian, a Generative Gaussian Splatting framework that produces high-quality textured meshes in just 2 minutes from a single-view image.

- Project: https://t.co/VtVO63AKG7
- Paper: https://t.co/ipIxZMS2SY
- Code: https://t.co/edPso1vo8O

[Boz @boztank]

Ray-Ban Meta smart glasses are live! They take great photos and 1080p video with the all-new 12MP camera. Speakers have 50% higher max volume and double the bass. And they come with a built-in Meta AI assistant (beta starting in US) and livestreaming to FB/IG. Still starting at $299! (Huge fan of the transparent frames myself)

[Ziwei Liu @liuziwei7]

We propose 🔥LaVie🔥, a high-quality text-to-video generation foundation model that operates on cascaded video latent diffusion models.

- Project: https://t.co/dTkQ66qsTe
- Paper: https://t.co/9UIG2Elv2f
- Code: https://t.co/sT24IvkJNu

[AK @_akhaliq]

FreeU: Free Lunch in Diffusion U-Net

paper page: https://t.co/fpRjbk0CED

we uncover the untapped potential of diffusion U-Net, which serves as a "free lunch" that substantially improves the generation quality on the fly. We initially investigate the key contributions of the U-Net architecture to the denoising process and identify that its main backbone primarily contributes to denoising, whereas its skip connections mainly introduce high-frequency features into the decoder module, causing the network to overlook the backbone semantics. Capitalizing on this discovery, we propose a simple yet effective method-termed "FreeU" - that enhances generation quality without additional training or finetuning. Our key insight is to strategically re-weight the contributions sourced from the U-Net's skip connections and backbone feature maps, to leverage the strengths of both components of the U-Net architecture. Promising results on image and video generation tasks demonstrate that our FreeU can be readily integrated to existing diffusion models, e.g., Stable Diffusion, DreamBooth, ModelScope, Rerender and ReVersion, to improve the generation quality with only a few lines of code.

[AK @_akhaliq]

Predicting masked tokens in stochastic locations improves masked image modeling

paper page: https://t.co/tQMmphD5Pm

Self-supervised learning is a promising paradigm in deep learning that enables learning from unlabeled data by constructing pretext tasks that require learning useful representations. In natural language processing, the dominant pretext task has been masked language modeling (MLM), while in computer vision there exists an equivalent called Masked Image Modeling (MIM). However, MIM is challenging because it requires predicting semantic content in accurate locations. E.g, given an incomplete picture of a dog, we can guess that there is a tail, but we cannot determine its exact location. In this work, we propose FlexPredict, a stochastic model that addresses this challenge by incorporating location uncertainty into the model. Specifically, we condition the model on stochastic masked token positions to guide the model toward learning features that are more robust to location uncertainties. Our approach improves downstream performance on a range of tasks, e.g, compared to MIM baselines, FlexPredict boosts ImageNet linear probing by 1.6% with ViT-B and by 2.5% for semi-supervised video segmentation using ViT-L.

[Fabian Mentzer @mentzer_f]

1/ Excited to share that our work on masked transformers for compression has been accepted at #ICCV2023! We show two things: a) how a simple transformer based architecture can lead to state-of-the-art image compression results: https://t.co/WnB74jsqKk

[Ziwei Liu @liuziwei7]

"Benchmarking and Analyzing Generative Data for Visual Recognition"

* GenBench: a broad benchmark on generative data for recognition
* CLER: a training-free metric on generative data's efficiency for recognition

- Paper: https://t.co/2QkAywomSV
- Code: https://t.co/dEMdotISW1

[AK @_akhaliq]

Interpolating between Images with Diffusion Models

paper page: https://t.co/h3WJiNDCmt

One little-explored frontier of image generation and editing is the task of interpolating between two input images, a feature missing from all currently deployed image generation pipelines. We argue that such a feature can expand the creative applications of such models, and propose a method for zero-shot interpolation using latent diffusion models. We apply interpolation in the latent space at a sequence of decreasing noise levels, then perform denoising conditioned on interpolated text embeddings derived from textual inversion and (optionally) subject poses. For greater consistency, or to specify additional criteria, we can generate several candidates and use CLIP to select the highest quality image. We obtain convincing interpolations across diverse subject poses, image styles, and image content, and show that standard quantitative metrics such as FID are insufficient to measure the quality of an interpolation.

[Peyman Milanfar @docmilanfar]

If you want to see great technical writing, read anything by @ddonoho11. I admire Dave for many reasons but for papers definitely read this one

"High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality" AMS math challenges lecture, 2000
https://t.co/WV4fqehVaN

[Creative.Edge CL+ @Mr_CreativeEdge]

Re:
試してみましたが凄いですね..
動画生成、どこまで進化するのか

・Midjourneyで画像を生成
・その画像をGen-2でビデオ生成

#Gen2 #imagetovideo

[AK @_akhaliq]

Generate Anything Anywhere in Any Scene

paper page: https://t.co/rxA46oMuh8

Text-to-image diffusion models have attracted considerable interest due to their wide applicability across diverse fields. However, challenges persist in creating controllable models for personalized object generation. In this paper, we first identify the entanglement issues in existing personalized generative models, and then propose a straightforward and efficient data augmentation training strategy that guides the diffusion model to focus solely on object identity. By inserting the plug-and-play adapter layers from a pre-trained controllable diffusion model, our model obtains the ability to control the location and size of each generated personalized object. During inference, we propose a regionally-guided sampling technique to maintain the quality and fidelity of the generated images. Our method achieves comparable or superior fidelity for personalized objects, yielding a robust, versatile, and controllable text-to-image diffusion model that is capable of generating realistic and personalized images. Our approach demonstrates significant potential for various applications, such as those in art, entertainment, and advertising design.

[AK @_akhaliq]

DreamDiffusion: Generating High-Quality Images from Brain EEG Signals

paper page: https://t.co/IN9nHz4q3m

paper introduces DreamDiffusion, a novel method for generating high-quality images directly from brain electroencephalogram (EEG) signals, without the need to translate thoughts into text. DreamDiffusion leverages pre-trained text-to-image models and employs temporal masked signal modeling to pre-train the EEG encoder for effective and robust EEG representations. Additionally, the method further leverages the CLIP image encoder to provide extra supervision to better align EEG, text, and image embeddings with limited EEG-image pairs. Overall, the proposed method overcomes the challenges of using EEG signals for image generation, such as noise, limited information, and individual differences, and achieves promising results. Quantitative and qualitative results demonstrate the effectiveness of the proposed method as a significant step towards portable and low-cost ``thoughts-to-image'', with potential applications in neuroscience and computer vision.