Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue System
Understanding dynamic scenes and dialogue contexts in order to converse with users has been challenging for multimodal dialogue systems. The 8-th Dialog System Technology Challenge (DSTC8) proposed an Audio Visual Scene-Aware Dialog (AVSD) task, which contains multiple modalities including audio, vision, and language, to evaluate how dialogue systems understand different modalities and response to users. In this paper, we proposed a multi-step joint-modality attention network (JMAN) based on recurrent neural network (RNN) to reason on videos. Our model performs a multi-step attention mechanism and jointly considers both visual and textual representations in each reasoning process to better integrate information from the two different modalities. Compared to the baseline released by AVSD organizers, our model achieves a relative 12.1% and 22.4% improvement over the baseline on ROUGE-L score and CIDEr score.
Code (0)
등록된 구현이 없습니다.
Tasks
Scene-Aware DialogueSimilar Papers 제목 키워드 기반
Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecasting
This paper presents a novel vehicle motion forecasting method based on multi-head attention. It produces joint forecasts for all vehicles on a road scene as sequences of multi-modal probability density functions of their…
Motion ForecastingPositionTrajectory ForecastingTrajectory Prediction for Autonomous Driving based on Multi-Head Attention with Joint Agent-Map Representation
Predicting the trajectories of surrounding agents is an essential ability for autonomous vehicles navigating through complex traffic scenes. The future trajectories of agents can be inferred using two important cues: the…
Autonomous DrivingAutonomous VehiclesTrajectory PredictionSGAligner++: Cross-Modal Language-Aided 3D Scene Graph Alignment
Aligning 3D scene graphs is a crucial initial step for several applications in robot navigation and embodied perception. Current methods in 3D scene graph alignment often rely on single-modality point cloud data and stru…
3D Scene Graph AlignmentVisual LocalizationScene Understanding3D Reconstruction3D Multi-Object Tracking Using Graph Neural Networks with Cross-Edge Modality Attention
Online 3D multi-object tracking (MOT) has witnessed significant research interest in recent years, largely driven by demand from the autonomous systems community. However, 3D offline MOT is relatively less explored. Labe…
3D Multi-Object TrackingGraph Neural NetworkMulti-Object TrackingObject TrackingMulti-Task Cross-Modality Attention-Fusion for 2D Object Detection
Accurate and robust object detection is critical for autonomous driving. Image-based detectors face difficulties caused by low visibility in adverse weather conditions. Thus, radar-camera fusion is of particular interest…
2D Object DetectionAutonomous DrivingObjectobject-detection+2