Explaining Multimodal Data Fusion: Occlusion Analysis for Wilderness Mapping
Jointly harnessing complementary features of multi-modal input data in a common latent space has been found to be beneficial long ago. However, the influence of each modality on the models decision remains a puzzle. This study proposes a deep learning framework for the modality-level interpretation of multimodal earth observation data in an end-to-end fashion. While leveraging an explainable machine learning method, namely Occlusion Sensitivity, the proposed framework investigates the influence of modalities under an early-fusion scenario in which the modalities are fused before the learning process. We show that the task of wilderness mapping largely benefits from auxiliary data such as land cover and night time light data.
Code (0)
등록된 구현이 없습니다.
Tasks
Earth ObservationSimilar Papers 제목 키워드 기반
CAMF-Det: Closure-Aware Multimodal Fusion for LiDAR-Camera 3D Object Detection on UAV Platforms
Multimodal 3D object detection based on LiDAR and cameras has demonstrated excellent performance in ground-vehicle scenarios, but has not been explored for Unmanned Aerial Vehicle (UAV) platforms. In UAV top-down scenes,…
3D Object DetectionData AugmentationAdaptive occlusion sensitivity analysis for visually explaining video recognition networks
This paper proposes a method for visually explaining the decision-making process of video recognition networks with a temporal extension of occlusion sensitivity analysis, called Adaptive Occlusion Sensitivity Analysis (…
Decision Makingimage-classificationImage ClassificationSensitivity+2MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis
Multimodal sentiment analysis relies on language, visual, and acoustic cues, but utterance-level modality quality may vary due to occlusion, background noise, motion blur, or imperfect transcripts, causing conventional f…
Multimodal Sentiment AnalysisDiffexplainer: Towards Cross-modal Global Explanations with Diffusion Models
We present DiffExplainer, a novel framework that, leveraging language-vision models, enables multimodal global explainability. DiffExplainer employs diffusion models conditioned on optimized text prompts, synthesizing im…
Explanations for Occluded Images
Existing algorithms for explaining the output of image classifiers perform poorly on inputs where the object of interest is partially occluded. We present a novel, black-box algorithm for computing explanations that uses…