Leveraging Topics and Audio Features with Multimodal Attention for Audio Visual Scene-Aware Dialog
With the recent advancements in Artificial Intelligence (AI), Intelligent Virtual Assistants (IVA) such as Alexa, Google Home, etc., have become a ubiquitous part of many homes. Currently, such IVAs are mostly audio-based, but going forward, we are witnessing a confluence of vision, speech and dialog system technologies that are enabling the IVAs to learn audio-visual groundings of utterances. This will enable agents to have conversations with users about the objects, activities and events surrounding them. In this work, we present three main architectural explorations for the Audio Visual Scene-Aware Dialog (AVSD): 1) investigating `topics' of the dialog as an important contextual feature for the conversation, 2) exploring several multimodal attention mechanisms during response generation, 3) incorporating an end-to-end audio classification ConvNet, AclNet, into our architecture. We discuss detailed analysis of the experimental results and show that our model variations outperform the baseline system presented for the AVSD task.
Code (0)
등록된 구현이 없습니다.
Tasks
Audio ClassificationResponse GenerationSimilar Papers 제목 키워드 기반
Exploring Context, Attention and Audio Features for Audio Visual Scene-Aware Dialog
We are witnessing a confluence of vision, speech and dialog system technologies that are enabling the IVAs to learn audio-visual groundings of utterances and have conversations with users about the objects, activities an…
Audio ClassificationVisual GroundingA Novel Multimodal Music Genre Classifier using Hierarchical Attention and Convolutional Neural Network
Music genre classification is one of the trending topics in regards to the current Music Information Retrieval (MIR) Research. Since, the dependency of genre is not only limited to the audio profile, we also make use of …
Genre classificationInformation RetrievalMusic Genre ClassificationMusic Information Retrieval+1Inconsistency-Aware Cross-Attention for Audio-Visual Fusion in Dimensional Emotion Recognition
Leveraging complementary relationships across modalities has recently drawn a lot of attention in multimodal emotion recognition. Most of the existing approaches explored cross-attention to capture the complementary rela…
Emotion RecognitionMultimodal Emotion RecognitionCosine Similarity of Multimodal Content Vectors for TV Programmes
Multimodal information originates from a variety of sources: audiovisual files, textual descriptions, and metadata. We show how one can represent the content encoded by each individual source using vectors, how to combin…
DiversityContext, Attention and Audio Feature Explorations for Audio Visual Scene-Aware Dialog
With the recent advancements in AI, Intelligent Virtual Assistants (IVA) have become a ubiquitous part of every home. Going forward, we are witnessing a confluence of vision, speech and dialog system technologies that ar…
Audio ClassificationGeneral Classification