VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Models
Large Vision-Language Models (LVLMs) have made remarkable developments along with the recent surge of large language models. Despite their advancements, LVLMs have a tendency to generate plausible yet inaccurate or inconsistent information based on the provided source content. This phenomenon, also known as ``hallucinations" can have serious downstream implications during the deployment of LVLMs. To address this, we present VORD a simple and effective method that alleviates hallucinations by calibrating token predictions based on ordinal relationships between modified image pairs. VORD is presented in two forms: 1.) a minimalist training-free variant which eliminates implausible tokens from modified image pairs, and 2.) a trainable objective function that penalizes unlikely tokens. Our experiments demonstrate that VORD delivers better calibration and effectively mitigates object hallucinations on a wide-range of LVLM benchmarks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
RevOrder: A Novel Method for Enhanced Arithmetic in Language Models
This paper presents RevOrder, a novel technique aimed at improving arithmetic operations in large language models (LLMs) by reversing the output digits in addition, subtraction, and n-digit by 1-digit (nD by 1D) multipli…
GSM8KMathFlavorDB2: An Updated Database of Flavor Molecules
Flavor is expressed through interaction of molecules via gustatory and olfactory mechanisms. Knowing the utility of flavor molecules in food and fragrances, it is valuable to add a comprehensive repository of flavor comp…
Coevolutionary Algorithm for Building Robust Decision Trees under Minimax Regret
In recent years, there has been growing interest in developing robust machine learning (ML) models that can withstand adversarial attacks, including one of the most widely adopted, efficient, and interpretable ML algorit…
Calibration of ordinal regression networks
Recent studies have shown that deep neural networks are not well-calibrated and often produce over-confident predictions. The miscalibration issue primarily stems from using cross-entropy in classifications, which aims t…
regressionFlavorDiffusion: Predicting Food Pairings and Chemical Interactions Using Diffusion Models
The study of food pairing has evolved beyond subjective expertise with the advent of machine learning. This paper presents FlavorDiffusion, a novel framework leveraging diffusion models to predict food-chemical interacti…