홈 › Papers

MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences

2020-10-22 · NAACL 2021 4 · Jianing Yang, Yongxin Wang, Ruitao Yi, Yuying Zhu, Azaan Rehman, Amir Zadeh, Soujanya Poria, Louis-Philippe Morency

Human communication is multimodal in nature; it is through multiple modalities such as language, voice, and facial expressions, that opinions and emotions are expressed. Data in this domain exhibits complex multi-relational and temporal interactions. Learning from this data is a fundamentally challenging research problem. In this paper, we propose Modal-Temporal Attention Graph (MTAG). MTAG is an interpretable graph-based neural model that provides a suitable framework for analyzing multimodal sequential data. We first introduce a procedure to convert unaligned multimodal sequence data into a graph with heterogeneous nodes and edges that captures the rich interactions across modalities and through time. Then, a novel graph fusion operation, called MTAG fusion, along with a dynamic pruning and read-out technique, is designed to efficiently process this modal-temporal graph and capture various interactions. By learning to focus only on the important interactions within the graph, MTAG achieves state-of-the-art performance on multimodal sentiment analysis and emotion recognition benchmarks, while utilizing significantly fewer model parameters.

📄 PDF Abstract BibTeX arXiv:2010.11985

Code (1)

jedyang97/MTAG 공식 구현 pytorch

Tasks

Emotion Recognition Multimodal Sentiment Analysis Sentiment Analysis

Methods 이 논문이 사용한 방법론

Pruning 설명 없음

Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusion

2020-11-27 · Sijie Mai, Songlong Xing, Jiaxuan He, Ying Zeng 외

In this paper, we study the task of multimodal sequence analysis which aims to draw inferences from visual, language and acoustic sequences. A majority of existing works generally focus on aligned fusion, mostly at word …

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

2021-12-03 · Ziwang Fu, Feng Liu, HanYang Wang, Siyuan Shen 외

Learning modality-fused representations and processing unaligned multimodal sequences are meaningful and challenging in multimodal emotion recognition. Existing approaches use directional pairwise attention or a message …

Efficient Neural Network Emotion Recognition Multimodal Emotion Recognition

Hengam: An Adversarially Trained Transformer for Persian Temporal Tagging

2022-11-20 · Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing 2022 11 · Sajad Mirzababaei, Amir Hossein Kargaran, Hinrich Schütze, Ehsaneddin Asgari

Many NLP main tasks benefit from an accurate understanding of temporal expressions, e.g., text summarization, question answering, and information retrieval. This paper introduces Hengam, an adversarially trained transfor…

Information Retrieval Named Entity Recognition (NER)Question Answering Retrieval+3

Graph-based Semantic Calibration Network for Unaligned UAV RGBT Image Semantic Segmentation and A Large-scale Benchmark

2026-04-29 · Fangqiang Fan, Zhicheng Zhao, Xiaoliang Ma, Chenglong Li 외 arxiv

Fine-grained RGBT image semantic segmentation is crucial for all-weather unmanned aerial vehicle (UAV) scene understanding. However, UAV RGBT image semantic segmentation faces two coupled challenges: cross-modal spatial …

Semantic Segmentation Scene Understanding

Graph Capsule Aggregation for Unaligned Multimodal Sequences

2021-08-17 · Jianfeng Wu, Sijie Mai, Haifeng Hu

Humans express their opinions and emotions through multiple modalities which mainly consist of textual, acoustic and visual modalities. Prior works on multimodal sentiment analysis mostly apply Recurrent Neural Network (…

Multimodal Sentiment Analysis Sentiment Analysis