paper-with-me

홈 › Papers

Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMs

2026-02-10 · Jingyi Wang, Fei Li, Rujie Liu arxiv

Existing Large Vision-Language Models (LVLMs) exhibit insufficient visual attention, leading to hallucinations. To alleviate this problem, some previous studies adjust and amplify visual attention. These methods present a limitation that boosting attention for all visual tokens inevitably increases attention to task irrelevant tokens. To tackle this challenge, we propose a training free attentional intervention algorithm to enhance the attention of task-relevant tokens based on the argument that task-relevant tokens generally demonstrate high visual-textual similarities. Specifically, the vision-text cross-attention submatrices, which represent visual-textual correlations, are extracted to construct the reweighting matrices to reallocate attention. Besides, to enhance the contribution of visual tokens, we inject visual attention values into the beam search decoding to identify solutions with higher visual attention. Extensive experiments demonstrate that this method significantly reduces hallucinations across mainstream LVLMs, while preserving the accuracy and coherence of generated content.

📄 PDF Abstract BibTeX arXiv:2602.09521

Code (0)

등록된 구현이 없습니다.

Similar Papers 제목 키워드 기반

Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models

2024-05-28 · Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi 외

This study addresses the issue observed in Large Vision Language Models (LVLMs), where excessive attention on a few image tokens, referred to as blind tokens, leads to hallucinatory responses in tasks requiring fine-grai…

MMEObject

Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation

2026-04-09 · Jiahao Li, Yang Lu, Yachao Zhang, Fangyong Wang 외 arxiv

Open-vocabulary semantic segmentation (OVSS) aims to segment arbitrary category regions in images using open-vocabulary prompts, necessitating that existing methods possess pixel-level vision-language alignment capabilit…

Semantic Segmentation

Retrieval-augmented Decoding for Improving Truthfulness in Open-ended Generation

2025-08-04 · Manh Nguyen, Sunil Gupta, Hung Le arxiv

Ensuring truthfulness in large language models (LLMs) remains a critical challenge for reliable text generation. While supervised fine-tuning and reinforcement learning with human feedback have shown promise, they requir…

Reinforcement LearningText Generation

When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs

2026-05-12 · Fanpu Cao, Xin Zou, Xuming Hu, Hui Xiong arxiv

Multimodal large language models (MLLMs) have become a key interface for visual reasoning and grounded question answering, yet they remain vulnerable to visual hallucinations, where generated responses contradict image c…

Question AnsweringVisual Reasoning

QuantWM: Temporally Consistent 2-Bit KV Cache Quantization for Video World Models

2026-09-28 · Jiaqi Zhao, Xiaobin Hu, Bo Yin, Junpeng Jiang 외 hf

Video world models achieve long-range temporal consistency by storing KV cache during generation, but the growing cache makes KV cache memory a major deployment bottleneck, which motivates low-bit quantization study for …