Piculet: Specialized Models-Guided Hallucination Decrease for MultiModal Large Language Models
Multimodal Large Language Models (MLLMs) have made significant progress in bridging the gap between visual and language modalities. However, hallucinations in MLLMs, where the generated text does not align with image content, continue to be a major challenge. Existing methods for addressing hallucinations often rely on instruction-tuning, which requires retraining the model with specific data, which increases the cost of utilizing MLLMs further. In this paper, we introduce a novel training-free method, named Piculet, for enhancing the input representation of MLLMs. Piculet leverages multiple specialized models to extract descriptions of visual information from the input image and combine these descriptions with the original image and query as input to the MLLM. We evaluate our method both quantitively and qualitatively, and the results demonstrate that Piculet greatly decreases hallucinations of MLLMs. Our method can be easily extended to different MLLMs while being universal.
Code (0)
등록된 구현이 없습니다.
Tasks
HallucinationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision
Large multimodal models suffer from multimodal hallucination, where they provide incorrect responses misaligned with the given visual information. Recent works have conjectured that one of the reasons behind multimodal h…
HallucinationMM-VetVisual Question AnsweringGuided Verifier: Collaborative Multimodal Reasoning via Dynamic Process Supervision
Reinforcement Learning (RL) has emerged as a pivotal mechanism for enhancing the complex reasoning capabilities of Multimodal Large Language Models (MLLMs). However, prevailing paradigms typically rely on solitary rollou…
Reinforcement LearningMultimodal ReasoningHallucination Detection-Guided Preference Optimization for Clinical Summarization
Large language models (LLMs) have shown promise on summarization tasks, but they often produce hallucinations, which are unsupported or incorrect statements that limit their reliability in specialized healthcare applicat…
IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination
Hallucination remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While Direct Preference Optimization (DPO) is a key alignment framework, existing approaches often rely heavily on costly extern…
V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimization
We introduce V-tableR1, a process-supervised reinforcement learning framework that elicits rigorous, verifiable reasoning from multimodal large language models (MLLMs). Current MLLMs trained solely on final outcomes ofte…
Reinforcement LearningVisual Reasoning