Top Tweets for #AutoHallusion

Vision-Language Models (VLMs) suffer from hallucinations👀🙈: their reasoning may ignore some objects or create non-existing ones. This is usually caused by their language model bias dominating the facts in the input visual signals, just like the human brain relying on memory🧠 and past experiences🌎 (called "schema" in cognitive science). But building a benchmark to evaluate and diagnose VLM hallucinations need a lot of human efforts #HallusionBench. Can we automatically generate (image, prompt) pairs that can induce hallucinations of VLMs? 🤔 The answer is Yes. We introduce #AutoHallusion🚀, an automatic hallucination pipeline that scales up the benchmark generation process. It probes LLMs to determine image content that contradicts with the LLM prior, and then generate the image by manipulating objects in the image using image-editing models. (1/n)👇

Vision-Language Models (VLMs) suffer from hallucinations👀🙈: their reasoning may ignore some objects or create non-existing ones. This is usually caused by their language model bias dominating the facts in the input visual signals, just like the human brain relying on memory🧠 and past experiences🌎 (called "schema" in cognitive science). But building a benchmark to evaluate and diagnose VLM hallucinations need a lot of human efforts #HallusionBench. Can we automatically generate (image, prompt) pairs that can induce hallucinations of VLMs? 🤔 The answer is Yes. We introduce #AutoHallusion🚀, an automatic hallucination pipeline that scales up the benchmark generation process. It probes LLMs to determine image content that contradicts with the LLM prior, and then generate the image by manipulating objects in the image using image-editing models. (1/n)👇