Kim Anthony

[Wonjae Kim @dandelin_kim]

@ai_den_lee @twelvelabs YouTube gets ~80 years of video uploaded per day (2023).

@twelvelabs just processed 17 years in a day. That's ~1/5 of YouTube's daily upload volume.

And we ran model inference on all of it.

[Aiden Lee @ai_den_lee]

17 years tokenized in a day

[Rosinality @rosinality]

Embedding layers. Adds a weighted sum of token embeddings for each layer. The motivation is interesting. Rare tokens are less trained, and this makes the hidden states for these tokens become too similar.

[Rosinality @rosinality]

This is my summary of a podcast interviewing Xiangyu Zhang (https://t.co/HPHCJuQnOh). I found this very insightful. As I have used whisper to transcribe and gemini to translate this, it could contain errors. Though, considering the overall flow of the content, I think it would be close to the original meaning.

Parentheses are my own comments.

## Unsupervised Learning for Images

Why hasn't unsupervised learning for images been as successful as for natural language? Contrastive Learning, for instance, did not yield good results at a large scale. Why?

These methods learn invariance. For example, they learn invariance to image transformations like cropping. However, the specific invariance to be learned is not derived from the data itself. It is handcrafted by humans. The same applies to Masked Image Modeling like MAE. MIM learns invariance to occlusion (parts of the image being masked). Even as the data scale increases, the class of invariance being learned remains fixed, and the model does not learn more kind of invariances.

In natural language, training to generate leads to understanding, generation, and human alignment all at once. Generation is a given since it's a generative model. When you provide context, it influences the subsequent output, which is a form of understanding. And since the training data is generated by humans, alignment with humans occurs naturally.

What about images? Similarly, let's assume we train a generative model that can extract the most information from images. Generation is possible. But does understanding emerge?

A generative model for images learns the distribution of pixels. There are various ways to model this pixel distribution, but only some of them would align with the human's understanding of images.

Images are data created by nature and are, in that sense, external to humans. (Of course, this wouldn't always be true. Photos reflect intent, and artificial objects can be subjects. But this will be limited.) Therefore, information about how humans understand an image is not inherent in the image itself. An image simply exists, separate from how humans understand it. This is why we inject invariance. It is an injection of human understanding of images. An alien might understand the same image in a completely different way than a human.

Therefore, for images, understanding and generation are disconnected. Naturally, there is no guarantee of alignment with humans.

(This is similar to something I recall, a comment from an OpenAI researcher, as far as I know. They expected that combining images and text for training would naturally improve performance, but it didn't. The suspected reason was that images are extremely concrete, while language is extremely abstract. For example, consider the text "children are playing." An image will depict a specific scene of children. But what does "playing" specifically look like? This shows that even a simple expression like "playing" is highly abstract.

This also relates to the limitations of image modeling. It's difficult to discover the human way of understanding an image from its characteristics alone.)

Video could be helpful, but it is difficult to handle and has low information density. Interleaved image-text data can be used, but it has significant limitations. Data like physical motion is almost non-existent. For these kinds of problems, we have no choice but to use video.

Nevertheless, in the short term, they trained models using image-text data, attempting to unify image understanding and generation. Understanding worked well, but generation did not. Furthermore, removing the generation component from training had no impact on understanding performance.

Suspecting it might be a data issue, they tried iterated improvement. Using the understanding model to create data, using that data to build a better generation model, and then integrating that generation model back into the understanding model for further training. They hoped this would eventually merge the two. However, they did not succeed. It worked, and they created more powerful understanding and generation models, but there was no synergy. Removing one did not affect the other.

Understanding and generation operated separately. For example, the model could immediately identify a physically impossible image, yet it would still generate physically impossible images. (Although it's often said there's a gap between generation and verification.)

## o1

o1 became a major turning point for this problem.

They discovered a strange phenomenon as model's size is scaled. The model's mathematical ability would improve, then plateau, and eventually decline. This was because the models developed a tendency to skip steps when solving problems. Data written by humans often omits intermediate steps. If there's data that omits calculations and only shows the final answer, alongside data that details the calculation process, a small model will find the former hard to learn, so the simpler, step-by-step latter pattern becomes dominant. However, a large model can learn to memorize the former pattern.

RL solves this problem, as skipping steps lowers the probability of getting the correct answer. Therefore, the pattern of skipping steps is suppressed.

Why does RL work for LLMs? Because of pretraining. Thanks to pretraining, even though the token sequence is very long, there are only a few critical decision points. The rest can be generated almost automatically. This drastically reduces the search space. The model only needs to find a few crucial branches.

Moreover, o1's reasoning process has a meta-CoT characteristic. At a critical forking point, it can be difficult to decide which branch to follow, especially for problems that cannot be solved in a single forward pass of transformer. So, what if the model tries one branch and then another? If one approach (CoT) fails, it can try another (Reflection).

This became a clue to the problem with image generation models. The problem with image generation is its immense complexity.

With autoregressive generation, once a part of an image is generated, it cannot be modified. Therefore, the model must decide how to generate an entire region in a single step. This exceeds the complexity a model can handle in one pass.

For instance, what if you ask a model to draw a blackboard and write out a complex mathematical problem and its solution? This task goes beyond simple image generation and has an underlying math problem. This clearly exceeds what a model can handle in one step.

Is diffusion different? Diffusion involves multiple steps, but it's ultimately a mathematically designed process of removing noise. The semantics are likely determined early in the generation process, with the rest of the steps just filling in details. Thus, it's not fundamentally different from an autoregressive model.

When training a diffusion model for image generation, it learns by sampling a specific timestep t. For most values of t, the diffusion process has little to do with understanding or semantics. After the semantics are set, rest is just filling the details. The learning signal from this process doesn't help with language understanding.

What happens when an image is generated without sufficient computational capacity? It's similar to training a text model under conditions of insufficient computation. For example, think about injecting into a pretraining model a bunch of problems and answers without the solution steps. The model either memorizes the answers or learns spurious correlations. The same would likely happen with images.

Therefore, current image generation models are stuck at the one-shot stage, where they must generate the answer in a single go. They lack not only meta-CoT but even a basic CoT.

So, they decided to first try implementing a CoT in the image space, starting with understanding. For a task like solving a maze, it's much easier if you can draw lines. They could train the model to solve the specific problems it was trained on, but it failed to generalize. This is in contrast to text CoT, where a model trained on math data can show generalization to a completely different domain.

Why did generalization work for text but not for images? A review of the pretraining text data reveals that while rare, there are examples of thought processes, such as verification or re-checking. However, the kind of data synthesized to train the image CoT does not exist in typical image-text datasets. Therefore, generalization through pre-training was not possible.

In models like o3, the CoT for images involves operations like cropping and zooming, likely because similar data exists on the internet.

So, how can this problem be solved? One way is to use video. Instructional videos, for example, contain many processes that can be considered thought processes. They show step-by-step progress, with actions like pointing with a laser pointer, sketching, and drawing lines. This could be helpful. However, video is difficult to handle. It's often too long, a lot of them are low-valued, and aligning it with text is hard.

Another way is to enable highly flexible image generation. This would allow the model to freely generate necessary images, independent of the original image. But if visual reasoning requires image generation, and image generation requires visual reasoning, isn't that a deadlock?

However, it seems possible to solve the image generation problem if the difficulty is well-controlled. Tasks like connect points A and B have low complexity and can be generated easily. The problem may have been that much image-text data is too difficult and was used for training without curation. Handling data well is, as always, crucial.

So, they built an image editing model. It could follow simple instructions well. When the ability to follow simple instructions is used and connected as needed for a task (CoT), visual reasoning problems will start to be solved. If understanding starts to solved like this, then, the generation problem will also be solved. Understanding means the final output is text, and generation just means the final output is an image.

## Long Context

The reason for needing a long context is that it's the easiest way to input a large amount of data into the model. However, current long context models can handle tasks like retrieval but struggle with complex reasoning. Furthermore, performance can decrease with longer context lengths, especially for smaller models.

In transformers, the storage capacity increases with the context length, so there is no incentive to compress the context. Moreover, training on retrieval tasks reinforces the tendency to preserve information. Is this beneficial for reasoning ability?

Alternatives like linear attention or state space models are being explored, but are they in the right direction? Often, eventually attention is used to solve the retrieval problem. Shouldn't retrieval be solved through approaches like tool use, model calling, or multi-agent systems?

With a multi-agent approach, you could have separate models for planning and execution. Nowadays, multi-agent systems can be trained end-to-end with RL.

## Autonomous and Online Learning

RL has been a powerful breakthrough for reasoning problems in language. However, RL has limitations and differs from human learning. Rule-based methods cannot be used if there are no easily calculable evaluation criteria. Using a model for evaluation can be hacked.

Humans, however, gain much of their feedback not from external sources but from their own judgment and awareness. Therefore, external evaluation alone is insufficient. Intrinsic motivation is needed.

Current methodologies will eventually hit a scaling problem. Not in model or data size, but in the need for environment scaling.

For example, to solve a programming problem, you must create an entire environment, including Docker containers and test data. All of this effort ultimately produces just one sample of data. Big techs are actually doing this.

But humans don't learn this way. Humans don't learn by experiencing countless handcrafted environments. Humans are self-driven. They read on their own, build their own environments, and learn from the feedback.

Achieving autonomous learning requires many things, but one of the most important is interaction with the environment. To interact properly, the problem of getting feedback from natural language must be solved.

Humans don't learn in a binary fashion. They receive feedback through natural language, which contains diverse dimensions and rich information. How would RL handle this feedback? It would create various evaluation criteria, assign weights, and sum them up. Much of the information in the feedback would be lost, reduced to just a single scalar.

There is no clear answer to this problem yet.

The challenges of autonomous learning include the long context problem, the problem of extracting information from natural language or unstructured feedback, and the problem of intrinsic reward, which guides the model to update itself to adapt to the world.

And autonomous learning would lead to online learning. A model capable of autonomous learning that then learns in a real environment would allow online learning. Imagine an autonomous agent operating within a company. It would become a powerful agent that has acquired all the company's internal information.

While multimodal reasoning might take about a year, autonomous and online learning might take about two to three years. Many people are already researching this problem, and it's a major topic in academia right now (like Ilya Sutskever).

## The Future

To create the next generation of technology, one must fully master the previous generation. Only then can its limitations and problems be fully understood.

The agent OpenAI talks about is likely different from the agents of previous generations. Consider OpenAI's 5 levels of AGI (Level 1: Conversational AI, Level 2: Reasoning AI, Level 3: Agentic AI, Level 4: Innovator AI, Level 5: Collective AI).

Moving from one level to the next required a new algorithm. Conversational AI could be achieved with next-token prediction. Reasoning AI required RL. What will be needed for agentic AI? Autonomous online learning. Agents require autonomy. They should not need humans to design countless environments and rules. Even reasoning models, once training is complete, cannot improve further, and their training depends on human-defined goals. However, in autonomous learning, the model must explore and learn on its own, setting its own goals. That will be the era of the agent.

Today's agents are still at the RL stage. The idea is to provide them with various tools and reward them for achieving a designer's desired outcome. Of course, this is better than the previous stage, where agents were pieced together with prompt engineering. Now, we are training the model to call tools on its own when it deems necessary, rather than having humans guide its actions.

[Beomyoung Kim @BeomyoungKim]

🚀Introducing ZIM: Zero-Shot Image Matting – A Step Beyond SAM (Segment Anything)!🚀

Please visit our project page: https://t.co/tB7aBJQ3U7

#ai #computervision #deeplearning #segmentation #matting

[Kim Anthony @KimAnth00773101]

Excited to share that our paper has been accepted at #ECCV2024!

We revitalize DenseNets and reveal their untapped potential !!

DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs - https://t.co/L34RXK5u6P

#RDNet #DenseNet #NAVER