When Correct Decisions Hide Internal Stress: Decision-State Probing in Multimodal Language Models
Multimodal language models are typically evaluated through external behavior: selecting the correct image--text match, rejecting unsupported captions, or answering visual queries correctly. However, correct behavior alone does not show that the model's internal decision state remains stable under controlled semantic stress. We study this gap through S$^3$E (Structured Semantic Stress Evaluation), a framework for analyzing behavior-internal decoupling in multimodal language models. S$^3$E uses a positive-anchored A/B forced-choice setup in which an image-supported caption is contrasted against semantic stress candidates under both original and swapped option orders, while hidden states are extracted at the pre-answer decision state. We focus on strict-correct trials, where the model consistently selects the correct caption across both orders. Rather than treating arbitrary hidden-state variation as evidence of instability, we measure whether semantic-conflict candidates induce excess decision-state displacement relative to meaning-preserving controls. Across Qwen3VL, Gemma3, and InternVL3, semantic stress consistently produces positive selected-layer excess displacement over lexical controls despite correct forced-choice behavior, while comparisons against random negatives are model-dependent. We interpret this as a scoped decision-state stress-sensitivity signal rather than evidence of downstream failure or hallucination. Our results suggest that forced-choice correctness alone is not a sufficient certificate of invariant internal decision geometry.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence
Many multimodal systems estimate the reliability of each modality and weight their contributions to the final prediction. However, it remains unclear whether these scores influence model decisions or merely correlate wit…
Sentiment AnalysisControlling for Confounders in Multimodal Emotion Classification via Adversarial Learning
Various psychological factors affect how individuals express emotions. Yet, when we collect data intended for use in building emotion recognition systems, we often try to do so by creating paradigms that are designed jus…
ClassificationEmotion ClassificationEmotion RecognitionGeneral ClassificationReal-Time Monitoring of User Stress, Heart Rate and Heart Rate Variability on Mobile Devices
Stress is considered to be the epidemic of the 21st-century. Yet, mobile apps cannot directly evaluate the impact of their content and services on user stress. We introduce the Beam AI SDK to address this issue. Using ou…
Heart rate estimationHeart Rate VariabilityMental Stress DetectionPhotoplethysmography (PPG)+2Operation-Adversarial Scenario Generation
This paper proposes a modified conditional generative adversarial network (cGAN) model to generate net load scenarios for power systems that are statistically credible, conditioned by given labels (e.g., seasons), and, a…
Generative Adversarial NetworkWhen Correct Beliefs Collapse: Epistemic Resilience of LLMs under Clinical Pressure
Despite strong medical benchmark accuracy, LLMs can exhibit severe multi-turn sycophancy in clinical dialogue, abandoning initial correct diagnosis under escalating pressure. We propose \textbf{\textsc{Med-Stress}}, a ta…