Context Sensitivity Improves Human-Machine Visual Alignment
Modern machine learning models typically represent inputs as fixed points in a high-dimensional embedding space. While this approach has been proven powerful for a wide range of downstream tasks, it fundamentally differs from the way humans process information. Because humans are constantly adapting to their environment, they represent objects and their relationships in a highly context-sensitive manner. To address this gap, we propose a method for context-sensitive similarity computation from neural network embeddings, applied to modeling a triplet odd-one-out task with an anchor image serving as simultaneous context. Modeling context enables us to achieve up to a 15% improvement in odd-one-out accuracy over a context-insensitive model. We find that this improvement is consistent across both original and "human-aligned" vision foundation models.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review
The intelligent dialogue system, aiming at communicating with humans harmoniously with natural language, is brilliant for promoting the advancement of human-machine interaction in the era of artificial intelligence. With…
A HVS-inspired Attention to Improve Loss Metrics for CNN-based Perception-Oriented Super-Resolution
Deep Convolutional Neural Network (CNN) features have been demonstrated to be effective perceptual quality features. The perceptual loss, based on feature maps of pre-trained CNN's has proven to be remarkably effective f…
Image RestorationSensitivitySuper-ResolutionSeeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visua…
Image ReconstructionSteering Multimodal Large Language Models Decoding for Context-Aware Safety
Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet their ability to make context-aware safety decisions remains limited. Existing methods often fail to balance oversensitiv…
PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment
Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored. Through repre…