Arantxa Casanova

Research Scientist at Meta, Generative AI

[Arantxa Casanova @arcp_8]

Check out our recent work "Instance-Conditioned GAN", a joint work with Marlene Careil, Jakob Verbeek, @adri_romsor and @michal_drozdzal.

We can generate images of unfamiliar scenes, such as camels in the snow!
Check out the details, paper and code in the blog post.

[AI at Meta @AIatMeta]

We’ve built a new image generation model called Instance-Conditioned GAN. Unlike traditional GANs, IC-GANs can create high-quality images of entirely unfamiliar objects and scenes that aren’t part of the model’s training data. Code+ details: https://t.co/SiaCpJW7oV

[Arantxa Casanova @arcp_8]

I am looking for PhD students for a 2024 summer internship at Meta in Generative AI: https://t.co/EeLpTsyYja
Specially relevant if interested in improving controllability and expressivity of image generative models, and want to work on challenging, domain-specific problems.

[Adriana Romero-Soriano @adri_romsor]

We are looking to hire FAIR research interns (current PhD students) in 2022.

Previous works led by students: https://t.co/si1C0dkJ8J, https://t.co/gPxAKeruef, https://t.co/Bt4qjsXt4B, https://t.co/mWpb7wsPk5, https://t.co/7OSEh1l9RR.

DM me or @michal_drozdzal with bio and CV.

[Replicate @replicate]

A demo of Instance-Conditioned GAN by @arcp_8 et al at @facebookai. It can generate images in a style of another image and can be conditioned by a class to alter the image.

https://t.co/6a52q6FnYD

[Arantxa Casanova @arcp_8]

@puneet2k @adri_romsor @michal_drozdzal (1/9)
Although both our papers condition the generator and discriminator on instance feature vectors, the methodology differs in terms of training, what we consider as “transfer”, and the presented results:

[Arantxa Casanova @arcp_8]

Check out our recent work "Instance-Conditioned GAN", a joint work with Marlene Careil, Jakob Verbeek, @adri_romsor and @michal_drozdzal.

We can generate images of unfamiliar scenes, such as camels in the snow!
Check out the details, paper and code in the blog post.

[AI at Meta @AIatMeta]

We’ve built a new image generation model called Instance-Conditioned GAN. Unlike traditional GANs, IC-GANs can create high-quality images of entirely unfamiliar objects and scenes that aren’t part of the model’s training data. Code+ details: https://t.co/SiaCpJW7oV

[Arantxa Casanova @arcp_8]

@puneet2k @adri_romsor @michal_drozdzal I enjoyed reading the paper and congrats on its acceptance. Thank you for pointing it out, we will cite it in the next updated version.
Although the pipeline may look similar at a first glance there are some important fundamental differences between the papers. Thread below:

[Arantxa Casanova @arcp_8]

@puneet2k @adri_romsor @michal_drozdzal (2/9)
Training:
- In DIP, the generated image obtained with the features C(x) is enforced to be close to the real image x. In IC-GAN, we condition on instance features (C(x) following your notation) but we never enforce the generated image to be close to the real image.

[Arantxa Casanova @arcp_8]

@puneet2k @adri_romsor @michal_drozdzal (3/9)
We want generated images to be in the distribution of real image nearest neighbors, enabling the generation of high quality, diverse samples. Matching a single real sample with a fake one, this diversity would be negatively impacted, as we saw in our early experimentation.

[Arantxa Casanova @arcp_8]

@puneet2k @adri_romsor @michal_drozdzal (4/9)
- DIP uses IMLE to match real-generated samples with nearest neighbors, while IC-GAN uses the standard GAN formulation to classify samples as real/fake.
- In DIP, only the features are used to condition the model, while IC-GAN can be conditioned on both features and class.

[Arantxa Casanova @arcp_8]

@puneet2k @adri_romsor @michal_drozdzal (5/9)
Key difference in transfer learning:
- DIP fine-tunes or trains from scratch in each of the datasets. IC-GAN performs zero-shot transfer: it is trained on ImageNet and then is directly transferred to other datasets without the need to re-train or fine-tune.

[Arantxa Casanova @arcp_8]

@puneet2k @adri_romsor @michal_drozdzal (6/9)
Done by conditioning on unseen instances from the “target” dataset at inference time. Training IC-GAN on ImageNet and performing inference on COCO-Stuff allows IC-GAN to obtain better quality and diversity generations, (Table 2 and Figure 3 in the paper, Supplementary F).

[Arantxa Casanova @arcp_8]

@puneet2k @adri_romsor @michal_drozdzal (7/9)
Results:
- We provide high resolution results (256x256) on ImageNet and COCO-Stuff (for both IC-GAN and class-conditional IC-GAN), surpassing the previous state of the art in the unlabeled scenario and getting on-pair results overall with BigGAN with more controllability.

[Arantxa Casanova @arcp_8]

@puneet2k @adri_romsor @michal_drozdzal (8/9)
- We showcase off-the-shelf transfer without the need to retrain or fine-tune.
- Our generations go beyond what is present in the training data or the conditioning instance when using both IC-GAN or the class-conditional IC-GAN (see the camel in the snow example).

[Arantxa Casanova @arcp_8]

@puneet2k @adri_romsor @michal_drozdzal (9/9)
For further details, we refer you to the paper: https://t.co/7dGrK01zqy.

[Arantxa Casanova @arcp_8]

@michael_rivard @facebookai The IC-GAN has been trained on ImageNet, where there is (1) images of camels, with practically only sand/vegetation backgrounds, and (2) images of other animals in the snow, such as bisons.

[AI at Meta @AIatMeta]

We’ve built a new image generation model called Instance-Conditioned GAN. Unlike traditional GANs, IC-GANs can create high-quality images of entirely unfamiliar objects and scenes that aren’t part of the model’s training data. Code+ details: https://t.co/SiaCpJW7oV

[Negar Rostamzadeh @negar_rz]

Check out our work on *Reinforced active learning for semantic segmentation* @iclr_conf presented by @arcp_8

Thursday 13:00-15:00 and 16:00-18:00 EDT
- Paper: https://t.co/6jvO71Casn
- Poster & slides: https://t.co/aeqJRVBgsC
- Code:https://t.co/myGVmuOdwc

[Adriana Romero-Soriano @adri_romsor]

Arantxa Casanova from @MILAMontreal presenting her work on iterative refinement of densely connected representation levels for semantic segmentation at #WiCV18 #CVPR18.

[David Vázquez @dvazquezcv]

@arcp_8 from @element_ai presenting her work from @MILAMontreal at #wad at #cvpr18