E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection
Multimodal image fusion and object detection are crucial for autonomous driving. While current methods have advanced the fusion of texture details and semantic information, their complex training processes hinder broader applications. Addressing this challenge, we introduce E2E-MFD, a novel end-to-end algorithm for multimodal fusion detection. E2E-MFD streamlines the process, achieving high performance with a single training phase. It employs synchronous joint optimization across components to avoid suboptimal solutions tied to individual tasks. Furthermore, it implements a comprehensive optimization strategy in the gradient matrix for shared parameters, ensuring convergence to an optimal fusion detection configuration. Our extensive testing on multiple public datasets reveals E2E-MFD's superior capabilities, showcasing not only visually appealing image fusion but also impressive detection outcomes, such as a 3.9% and 2.0% mAP50 increase on horizontal object detection dataset M3FD and oriented object detection dataset DroneVehicle, respectively, compared to state-of-the-art approaches. The code is released at https://github.com/icey-zhang/E2E-MFD.
Code (1)
Tasks
Autonomous DrivingObjectobject-detectionObject DetectionOriented Object DetectionSimilar Papers 제목 키워드 기반
FGU3R: Fine-Grained Fusion via Unified 3D Representation for Multimodal 3D Object Detection
Multimodal 3D object detection has garnered considerable interest in autonomous driving. However, multimodal detectors suffer from dimension mismatches that derive from fusing 3D points with 2D pixels coarsely, which lea…
3D Object DetectionAutonomous Drivingmultimodal interactionobject-detection+1Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Analyzing human multimodal language is an emerging area of research in NLP. Intrinsically this language is multimodal (heterogeneous), sequential and asynchronous; it consists of the language (words), visual (expressions…
Emotion RecognitionLanguage ModelingLanguage ModellingMultimodal Sentiment Analysis+2SODFormer: Streaming Object Detection with Transformer Using Events and Frames
DAVIS camera, streaming two complementary sensing modalities of asynchronous events and frames, has gradually been used to address major object detection challenges (e.g., fast motion blur and low-light). However, how to…
object-detectionObject DetectionChangeBridge: Spatiotemporal Image Generation with Multimodal Controls for Remote Sensing
Spatiotemporal image generation is a highly meaningful task, which can generate future scenes conditioned on given observations. However, existing change generation methods can only handle event-driven changes (e.g., new…
Change DetectionImage GenerationAsynchronous Multimodal Video Sequence Fusion via Learning Modality-Exclusive and -Agnostic Representations
Understanding human intentions (e.g., emotions) from videos has received considerable attention recently. Video streams generally constitute a blend of temporal data stemming from distinct modalities, including natural l…