SIMMC: Situated Interactive Multi-Modal Conversational Data Collection And Evaluation Platform
As digital virtual assistants become ubiquitous, it becomes increasingly important to understand the situated behaviour of users as they interact with these assistants. To this end, we introduce SIMMC, an extension to ParlAI for multi-modal conversational data collection and system evaluation. SIMMC simulates an immersive setup, where crowd workers are able to interact with environments constructed in AI Habitat or Unity while engaging in a conversation. The assistant in SIMMC can be a crowd worker or Artificial Intelligent (AI) agent. This enables both (i) a multi-player / Wizard of Oz setting for data collection, or (ii) a single player mode for model / system evaluation. We plan to open-source a situated conversational data-set collected on this platform for the Conversational AI research community.
Code (0)
등록된 구현이 없습니다.
Tasks
AI AgentUnityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Situated and Interactive Multimodal Conversations
Next generation virtual assistants are envisioned to handle multimodal inputs (e.g., vision, memories of previous interactions, in addition to the user's utterances), and perform multimodal actions (e.g., displaying a ro…
Response GenerationAn Analysis of State-of-the-Art Models for Situated Interactive MultiModal Conversations (SIMMC)
There is a growing interest in virtual assistants with multimodal capabilities, e.g., inferring the context of a conversation through scene understanding. The recently released situated and interactive multimodal convers…
Scene UnderstandingcPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers
An emerging area of research in situated and multimodal interactive conversations (SIMMC) includes interactions in scientific papers. Since scientific papers are primarily composed of text, equations, figures, and tables…
Which One Are You Referring To? Multimodal Object Identification in Situated Dialogue
The demand for multimodal dialogue systems has been rising in various domains, emphasizing the importance of interpreting multimodal inputs from conversational and situational contexts. We explore three methods to tackle…
Learning to Embed Multi-Modal Contexts for Situated Conversational Agents
The Situated Interactive Multi-Modal Conversations (SIMMC) 2.0 aims to create virtual shopping assistants that can accept complex multi-modal inputs, i.e. visual appearances of objects and user utterances. It consists of…
coreference-resolutionCoreference ResolutionDecoderdialog state tracking+2