Have the VLMs Lost Confidence? A Study of Sycophancy in VLMs
In the study of LLMs, sycophancy represents a prevalent hallucination that poses significant challenges to these models. Specifically, LLMs often fail to adhere to original correct responses, instead blindly agreeing with users' opinions, even when those opinions are incorrect or malicious. However, research on sycophancy in visual language models (VLMs) has been scarce. In this work, we extend the exploration of sycophancy from LLMs to VLMs, introducing the MM-SY benchmark to evaluate this phenomenon. We present evaluation results from multiple representative models, addressing the gap in sycophancy research for VLMs. To mitigate sycophancy, we propose a synthetic dataset for training and employ methods based on prompts, supervised fine-tuning, and DPO. Our experiments demonstrate that these methods effectively alleviate sycophancy in VLMs. Additionally, we probe VLMs to assess the semantic impact of sycophancy and analyze the attention distribution of visual tokens. Our findings indicate that the ability to prevent sycophancy is predominantly observed in higher layers of the model. The lack of attention to image knowledge in these higher layers may contribute to sycophancy, and enhancing image attention at high layers proves beneficial in mitigating this issue.
Code (0)
등록된 구현이 없습니다.
Tasks
HallucinationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models
Vision-language models (VLMs) adapted to the medical domain have shown strong performance on visual question answering benchmarks, yet their robustness against two critical failure modes, hallucination and sycophancy, re…
Visual Question AnsweringTowards Analyzing and Mitigating Sycophancy in Large Vision-Language Models
Large Vision-Language Models (LVLMs) have shown significant capability in vision-language understanding. However, one critical issue that persists in these models is sycophancy, which means models are unduly influenced b…
HallucinationPrompt EngineeringEchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models
Recent benchmarks for medical Large Vision-Language Models (LVLMs) emphasize leaderboard accuracy, overlooking reliability and safety. We study sycophancy -- models' tendency to uncritically echo user-provided informatio…
Accounting for Sycophancy in Language Model Uncertainty Estimation
Effective human-machine collaboration requires machine learning models to externalize uncertainty, so users can reflect and intervene when necessary. For language models, these representations of uncertainty may be impac…
Language ModelingLanguage ModellingmodelQuestion AnsweringBenchmarking and Mitigating Sycophancy in Medical Vision Language Models
Visual language models (VLMs) have the potential to transform medical workflows. However, the deployment is limited by sycophancy. Despite this serious threat to patient safety, a systematic benchmark remains lacking. Th…
Visual Question Answering