Exploring Large Language Models for Multi-Modal Out-of-Distribution Detection
Out-of-distribution (OOD) detection is essential for reliable and trustworthy machine learning. Recent multi-modal OOD detection leverages textual information from in-distribution (ID) class names for visual OOD detection, yet it currently neglects the rich contextual information of ID classes. Large language models (LLMs) encode a wealth of world knowledge and can be prompted to generate descriptive features for each class. Indiscriminately using such knowledge causes catastrophic damage to OOD detection due to LLMs' hallucinations, as is observed by our analysis. In this paper, we propose to apply world knowledge to enhance OOD detection performance through selective generation from LLMs. Specifically, we introduce a consistency-based uncertainty calibration method to estimate the confidence score of each generation. We further extract visual objects from each image to fully capitalize on the aforementioned world knowledge. Extensive experiments demonstrate that our method consistently outperforms the state-of-the-art.
Code (0)
등록된 구현이 없습니다.
Tasks
DescriptiveOut-of-Distribution DetectionOut of Distribution (OOD) DetectionWorld KnowledgeSimilar Papers 제목 키워드 기반
Stochastic Gradient Monomial Gamma Sampler
Recent advances in stochastic gradient techniques have made it possible to estimate posterior distributions from large datasets via Markov Chain Monte Carlo (MCMC). However, when the target posterior is multimodal, mixin…
Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysis
Recent advancements have enhanced the capability of Multimodal Large Language Models (MLLMs) to comprehend multi-image information. However, existing benchmarks primarily evaluate answer correctness, overlooking whether …
From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking
The rapid development of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) has exposed vulnerabilities to various adversarial attacks. This paper provides a comprehensive overview of jailbreaking …
Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space
Multimodal video sentiment analysis aims to integrate multiple modal information to analyze the opinions and attitudes of speakers. Most previous work focuses on exploring the semantic interactions of intra- and inter-mo…
Multimodal Sentiment AnalysisSentiment AnalysisExploring Machine Learning and Language Models for Multimodal Depression Detection
This paper presents our approach to the first Multimodal Personality-Aware Depression Detection Challenge, focusing on multimodal depression detection using machine learning and deep learning models. We explore and compa…