POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentation
Existing LVLM-based reasoning segmentation methods often suffer from imprecise segmentation results and hallucinations in their text responses. This paper introduces POPEN, a novel framework designed to address these issues and achieve improved results. POPEN includes a preference-based optimization method to finetune the LVLM, aligning it more closely with human preferences and thereby generating better text responses and segmentation results. Additionally, POPEN introduces a preference-based ensemble method for inference, which integrates multiple outputs from the LVLM using a preference-score-based attention mechanism for refinement. To better adapt to the segmentation task, we incorporate several task-specific designs in our POPEN framework, including a new approach for collecting segmentation preference data with a curriculum learning mechanism, and a novel preference optimization loss to refine the segmentation capability of the LVLM. Experiments demonstrate that our method achieves state-of-the-art performance in reasoning segmentation, exhibiting minimal hallucination in text responses and the highest segmentation accuracy compared to previous advanced methods like LISA and PixelLM.
Code (0)
등록된 구현이 없습니다.
Tasks
HallucinationReasoning SegmentationSegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning
Recently, large models have shown significant potential for smart healthcare. However, the deployment of Large Vision-Language Models (LVLMs) for clinical services is currently hindered by three critical challenges: a te…
Visual Question AnsweringImproving Generalization in Visual Reasoning via Self-Ensemble
The cognitive faculty of visual reasoning necessitates the integration of multimodal perceptual processing and commonsense and external knowledge of the world. In recent years, a plethora of large vision-language models …
Visual Question Answering (VQA)Visual ReasoningMMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization
The advancement of Large Vision-Language Models (LVLMs) has propelled their application in the medical field. However, Medical LVLMs (Med-LVLMs) encounter factuality challenges due to modality misalignment, where the mod…
Visual Question Answering (VQA)Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization
Despite recent advances in Large Video Language Models (LVLMs), they still struggle with fine-grained temporal understanding, hallucinate, and often make simple mistakes on even simple video question-answering tasks, all…
HallucinationQuestion AnsweringVideo Question AnsweringVideo UnderstandingVaPR -- Vision-language Preference alignment for Reasoning
Preference finetuning methods like Direct Preference Optimization (DPO) with AI-generated feedback have shown promise in aligning Large Vision-Language Models (LVLMs) with human preferences. However, existing techniques …
Response Generation