Nikolay Jetchev

Principal applied scientist @Zalando, deep learning expert, digital art experimenter.
See also

[Nikolay Jetchev @NJetchev]

"Armored Knight from the Orient"

I am getting better at shiny metal surfaces. With this prompt, CLIPMatrix created the 3D knight with beautiful surface engravings.

Would be curious to hear the opinion of some Blender pro - how quickly can a human artist do this?

#AIart #3dart

[Nikolay Jetchev @NJetchev]

@fablesimulation @Scobleizer Awesome, wanna try

[Nikolay Jetchev @NJetchev]

Virtual Try-on Research is fun, i hope that one day it will work even with armor and exotic stuff.

Kling AI with Elements function is indeed amazing. It is a whole different league than the basic Kling 1.6 version.

Fun times to make content with genAI

[Nikolay Jetchev @NJetchev]

https://t.co/ycNzmSDWZS

We're hiring! Principal Applied Scientist - Generative AI (Computer Vision). Join our creative research team to innovate and build cutting-edge AI models in the fashion industry. Apply now to shape the future of fashion tech!

#aigeneration #deeplearn

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

What’s the easiest way to specialize an LLM over your own data? Recent research has studied this problem in depth, and RAG is way more effective (and easier to implement) compared to extended pretraining or finetuning…

Knowledge from pretraining. A lot of factual information is inherently present within an LLM’s pretrained weights, but the knowledge possessed by these models is highly dependent upon the characteristics of their pretraining data. Unfortunately, this means that—at least in the current paradigm of LLMs—the knowledge base of these models is static (e.g., ChatGPT has a knowledge cutoff date) and may lack detailed information.

Knowledge injection. Given a pretrained LLM, there are two postprocessing techniques that we can use for injecting new data into the LLM’s knowledge base:

- Finetuning: continuing the model’s pretraining process over a smaller, domain-specialized corpus of new information.
- Retrieval Augmented Generation (RAG): modifying the LLM’s input query by retrieving relevant information that can be leveraged by the model via in-context learning to generate a more grounded/factual output.

The variant of finetuning referenced above is a continued pretraining style of finetuning, where a next token prediction objective is used to further train a pretrained model over a specialized corpus of text. In contrast, SFT and RLHF emphasize the quality of model responses rather than improving the LLM’s breadth of knowledge.

“Given some knowledge base in the form of a text corpus, what is the best way to teach a pre-trained model this knowledge?” - from [1]

Recent research. In [1], authors compare RAG and finetuning to determine the superior knowledge injection approach. The RAG setup uses vector search to retrieve relevant document chunks to include in the model’s prompt. Given a corpus of information, we can:

1. Divide this corpus into chunks of text.
2. Use an embedding model (e.g., bge-large-en) to generate a dense vector for each chunk of text.
3. Search for relevant chunks by embedding the model’s input and performing a vector search.
4. Add relevant chunk’s into the model’s prompt.

What do we learn? While finetuning does improve model performance, RAG consistently outperforms finetuning for the injection of both new and previously encountered knowledge. Put simply, LLMs struggle to learn new information through finetuning. Though finetuning does yield a benefit in performance relative to the base model, RAG has a significant advantage over finetuning. Combining RAG with finetuning—though effective in some cases—does not consistently benefit performance.

Finetuning with paraphrases. We can improve the performance of finetuning for knowledge injection by training the model over several different paraphrases of the same information. In order to teach an LLM new information via finetuning, we must repeat this information in numerous ways.

——
[1] Ovadia, Oded, et al. "Fine-tuning or retrieval? comparing knowledge injection in llms." arXiv preprint arXiv:2312.05934 (2023).

[Nikolay Jetchev @NJetchev]

"Oni Ninja in Hellish Future Tokyo, by Hieronymus Bosch"
Created using several new AI tools i learned recently, enjoy!

https://t.co/ykLoLnLgiN for modified SDXL

https://t.co/uzrfsbzzR7 for depth estimation

https://t.co/5oWwQvuKUO for depth and image animation

#AIart #anime

[Nikolay Jetchev @NJetchev]

"The Cyborg Oni Ninjas in Futuristic Tokyo"

I recommend opening the full size, to enjoy all the details of this gigantic seamless mosaic of size 3600x3600.

I worked on a lot of tricks to make panel blends seamless, here it worked nice with 4x4 fitting characters

#AIart #anime

[Nikolay Jetchev @NJetchev]

Explore the generative Tribal Totems Bestiary 🌿🐾

Behold a vast 3600x3600px mosaic! 🎨 My coding journey with SDXL & ChatGPT has crafted intricate collages. Discover 16 animals across 4 habitats, intricately woven into a vibrant tribal tapestry. 🐅🌊🦅🏞️

#AIart #DigitalMosaic

[Nikolay Jetchev @NJetchev]

"The Cybernetic Totem "
SDXL 1.0 outpainting

I like the functionality that comes with the base diffusion model, and customized the code to make seamless mosaics of size 4096x4096px and larger.

I am exploring what this tool can make at scale, ideas in the replies welcome.
#AIart

[Nikolay Jetchev @NJetchev]

The flow of thought between GPT4 and Dalle3 is amazing. Longer discussions on a prompt theme can lead to amazing discoveries in prompts and generated images.

A fun process to do on a Halloween night...

" Anime Hero with Oni Mask"

#AIart #AnimeArt

[Yann LeCun @ylecun]

Altman, Hassabis, and Amodei are the ones doing massive corporate lobbying at the moment.
They are the ones who are attempting to perform a regulatory capture of the AI industry.
You, Geoff, and Yoshua are giving ammunition to those who are lobbying for a ban on open AI R&D.

If your fear-mongering campaigns succeed, they will *inevitably* result in what you and I would identify as a catastrophe: a small number of companies will control AI.

The vast majority of our academic colleagues are massively in favor of open AI R&D. Very few believe in the doomsday scenarios you have promoted.
You, Yoshua, Geoff, and Stuart are the singular-but-vocal exceptions.

like many, I very much support open AI platforms because I believe in a combination of forces: people's creativity, democracy, market forces, and product regulations.
I also know that producing AI systems that are safe and under our control is possible. I've made concrete proposals to that effect.
This will all drive people to do the Right Thing.

You write as if AI is just happening, as if it were some natural phenomenon beyond our control.
But it's not. It's making progress because of individual people that you and I know. We, and they, have agency in building the Right Things.
Asking for regulation of R&D (as opposed to product deployment) implicitly assumes that these people and the organization they work for are incompetent, reckless, self-destructive, or evil. They are not.

I have made lots of arguments that the doomsday scenarios you are so afraid of are preposterous. I'm not going to repeat them here. But the main point is that if powerful AI systems are driven by objectives (which include guardrails) they will be safe and controllable because *e* set those guardrails and objectives.
(Current Auto-Regressive LLMs are not driven by objectives, so let's not extrapolate from their current weaknesses).

Now about open source: your campaign is going to have the exact opposite effect of what you seek.
In a future where AI systems are poised to constitute the repository of all human knowledge and culture, we *need* the platforms to be open source and freely available so that everyone can contribute to them.
Openness is the only way to make AI platforms reflect the entirety of human knowledge and culture.
This requires that contributions to those platforms be crowd-sourced, a bit like Wikipedia.
That won't work unless the platforms are open.

The alternative, which will *inevitably* happen if open source AI is regulated out of existence, is that a small number of companies from the West Coast of the US and China will control AI platform and hence control people's entire digital diet.
What does that mean for democracy?
What does that mean for cultural diversity?

*THIS* is what keeps me up at night.

[Nikolay Jetchev @NJetchev]

In light of recent news, I delved into a discussion with ChatGPT on civilization vs. barbarism and humanity's balance between the two. Astounded by how the language model crafts profound prompts and DALL·E turns them into stunning visuals.

Enjoy 4 images from that 🎨 #AIart

[Nikolay Jetchev @NJetchev]

I finally got access to Dalle3 inside chatGPT - and it is amazing! Free style of chats which could become image prompts immediately - it is a dream come true.

An example here, I chatted about Japanese traditional art, Oni masks and giant skeletons playing Kabuki theatre

#AIart

[Nikolay Jetchev @NJetchev]

I like the power of Dalle3 to generate images, using Bing Chat to make the text prompts.

I could tell the system "Make prompts for images for the history of the battle of Agincourt" and it would make diverse texts and images for them, which tell a powerful visual story.
#AIart

[Nikolay Jetchev @NJetchev]

Enjoy these Oni Masks, they have a certain aura even with a low-tech art look.

https://t.co/U48tDQxoKV is another cool StableDiffusion version, it is beatiful way to make ASCII art.

But things get even better if you mix with other LoRA modules, infinite possibilities.

#AIart

[Remi @remi_molettee]

Probably the most impressive thing I've tested this year. One thing's for sure: it's addictive!
#AnimateDiff

[Lior Alexander @LiorOnAI]

Most impressive paper I've seen this week.

Generative Image Dynamics transforms still images into videos or interactive scenes.

The Google team trained the model by using a dataset of motion trajectories from real-life videos of natural, oscillating motions like those seen in trees, flowers, candles, and wind-blown clothing.

Web: https://t.co/EbE9SibAZL

[apolinario (poli) @multimodalart]

Thanks @angrypenguinPNG for merging my PR to add high resolution to the Illusion Diffusion Space 📺🌀

It's now as fast, double the resolution and has crispy details - go play ▶️

https://t.co/FDoEOGfKnF

[Nikolay Jetchev @NJetchev]

"Autoportrait made of Medieval Town, Bruegel style"

The QR-code ControlNet for Stable Diffusion is so fun! The best AI mosaic tool ever used - and i have played with these and developed my own in the last 8 years.
My dream of making a digital Archimboldo is ever closer

#AIart

[Nikolay Jetchev @NJetchev]

In addition to digital art, i also like a lot pen art and ink drawings. Boris Pramatarov is one of my favorite artists.

In his honor, i brought to the 3d world an input 2d drawing from him, enjoy this surreal butterfly-girl

AI method: Magic123

#aiart #InkDrawing #3d