Frame Aggregation and Multi-Modal Fusion Framework for Video-Based Person Recognition

2020-10-19 · Fangtao Li, Wenzhe Wang, Zihe Liu, Haoran Wang, Chenghao Yan, Bin Wu

Video-based person recognition is challenging due to persons being blocked and blurred, and the variation of shooting angle. Previous research always focused on person recognition on still images, ignoring similarity and continuity between video frames. To tackle the challenges above, we propose a novel Frame Aggregation and Multi-Modal Fusion (FAMF) framework for video-based person recognition, which aggregates face features and incorporates them with multi-modal information to identify persons in videos. For frame aggregation, we propose a novel trainable layer based on NetVLAD (named AttentionVLAD), which takes arbitrary number of features as input and computes a fixed-length aggregation feature based on feature quality. We show that introducing an attention mechanism to NetVLAD can effectively decrease the impact of low-quality frames. For the multi-model information of videos, we propose a Multi-Layer Multi-Modal Attention (MLMA) module to learn the correlation of multi-modality by adaptively updating Gram matrix. Experimental results on iQIYI-VID-2019 dataset show that our framework outperforms other state-of-the-art methods.

📄 PDF Abstract BibTeX arXiv:2010.09290

Code (0)

등록된 구현이 없습니다.

Tasks

Person Recognition

FedUAF: Uncertainty-Aware Fusion with Reliability-Guided Aggregation for Multimodal Federated Sentiment Analysis

2026-02-28 · Xianxun Zhu, Zezhong Sun, Imad Rida, Erik Cambria 외 arxiv

Multimodal sentiment analysis in federated learning environments faces significant challenges due to missing modalities, heterogeneous data distributions, and unreliable client updates. Existing federated approaches ofte…

Multimodal Sentiment Analysis Federated Learning

CAMPA: Efficient and Aligned Multimodal Graph Learning via Decoupled Propagation and Aggregation

2026-05-12 · Daohan Su, Hao Liu, Xunkai Li, Yinlin Zhu 외 arxiv

Multimodal Graph Neural Networks (MGNNs) have shown strong potential for learning from multimodal attributed graphs, yet most existing approaches rely on tightly coupled architectures that suffer from prohibitive computa…

Representation Learning Graph Learning

Diffusion Models For Multi-Modal Generative Modeling

2024-07-24 · Changyou Chen, Han Ding, Bunyamin Sisman, Yi Xu 외

Diffusion-based generative modeling has been achieving state-of-the-art results on various generation tasks. Most diffusion models, however, are limited to a single-generation modeling. Can we generalize diffusion models…

Decoder Denoising multimodal generation

Federated Dialogue-Semantic Diffusion for Emotion Recognition under Incomplete Modalities

2025-11-01 · Xihang Qiu, Jiarong Cheng, Yuhao Fang, Wanpeng Zhang 외 arxiv

Multimodal Emotion Recognition in Conversations (MERC) enhances emotional understanding through the fusion of multimodal signals. However, unpredictable modality absence in real-world scenarios significantly degrades the…

Multimodal Emotion Recognition Emotion Classification Federated Learning

M^3Detection: Multi-Frame Multi-Level Feature Fusion for Multi-Modal 3D Object Detection with Camera and 4D Imaging Radar

2025-10-31 · Xiaozhi Li, Huijun Di, Jian Li, Feng Liu 외 arxiv

Recent advances in 4D imaging radar have enabled robust perception in adverse weather, while camera sensors provide dense semantic information. Fusing the these complementary modalities has great potential for cost-effec…

Computational Efficiency 3D Object Detection