Magnifier Prompt: Tackling Multimodal Hallucination via Extremely Simple Instructions
Hallucinations in multimodal large language models (MLLMs) hinder their practical applications. To address this, we propose a Magnifier Prompt (MagPrompt), a simple yet effective method to tackle hallucinations in MLLMs via extremely simple instructions. MagPrompt is based on the following two key principles, which guide the design of various effective prompts, demonstrating robustness: (1) MLLMs should focus more on the image. (2) When there are conflicts between the image and the model's inner knowledge, MLLMs should prioritize the image. MagPrompt is training-free and can be applied to open-source and closed-source models, such as GPT-4o and Gemini-pro. It performs well across many datasets and its effectiveness is comparable or even better than more complex methods like VCD. Furthermore, our prompt design principles and experimental analyses provide valuable insights into multimodal hallucination.
Code (0)
등록된 구현이 없습니다.
Tasks
HallucinationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Magnifier: A Multi-grained Neural Network-based Architecture for Burned Area Delineation
In crisis management and remote sensing, image segmentation plays a crucial role, enabling tasks like disaster response and emergency planning by analyzing visual data. Neural networks are able to analyze satellite acqui…
Burned Area DelineationDisaster ResponseImage SegmentationSemantic SegmentationMedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosis
Chest X-ray images are commonly used for predicting acute and chronic cardiopulmonary conditions, but efforts to integrate them with structured clinical data face challenges due to incomplete electronic health records (E…
Medical DiagnosisMedical Visual Question AnsweringVisual GroundingVisual Question Answering+2OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination
Recently, Omni-modal large language models (OLLMs) have sparked a new wave of research, achieving impressive results in tasks such as audio-video understanding and real-time environment perception. However, hallucination…
Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding
Existing vision-language models (VLMs) often suffer from visual hallucination, where the generated responses contain inaccuracies that are not grounded in the visual input. Efforts to address this issue without model fin…
HallucinationText GenerationVIGIL: Tackling Hallucination Detection in Image Recontextualization
We introduce VIGIL (Visual Inconsistency & Generative In-context Lucidity), the first benchmark dataset and framework providing a fine-grained categorization of hallucinations in the multimodal image recontextualization …