MONAH: Multi-Modal Narratives for Humans to analyze conversations
In conversational analyses, humans manually weave multimodal information into the transcripts, which is significantly time-consuming. We introduce a system that automatically expands the verbatim transcripts of video-recorded conversations using multimodal data streams. This system uses a set of preprocessing rules to weave multimodal annotations into the verbatim transcripts and promote interpretability. Our feature engineering contributions are two-fold: firstly, we identify the range of multimodal features relevant to detect rapport-building; secondly, we expand the range of multimodal annotations and show that the expansion leads to statistically significant improvements in detecting rapport-building.
Code (1)
Tasks
Emotion Recognition in ConversationFeature EngineeringText GenerationSimilar Papers 제목 키워드 기반
Detecting depression in dyadic conversations with multimodal narratives and visualizations
Conversations contain a wide spectrum of multimodal information that gives us hints about the emotions and moods of the speaker. In this paper, we developed a system that supports humans to analyze conversations. Our mai…
Humans vs Vision-Language Models: A Unified Measure of Narrative Coherence
We study narrative coherence in visually grounded stories by comparing human-written narratives with those generated by vision-language models (VLMs) on the Visual Writing Prompts corpus. Using a set of metrics that capt…
Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytelling
The paper proposes a framework that combines behavioral and computational experiments employing fictional prompts as a novel tool for investigating cultural artifacts and social biases in storytelling both by humans and …
SNAG: Spoken Narratives and Gaze Dataset
Humans rely on multiple sensory modalities when examining and reasoning over images. In this paper, we describe a new multimodal dataset that consists of gaze measurements and spoken descriptions collected in parallel du…
Conflicting narratives and polarization on social media
Narratives are key interpretative devices by which humans make sense of political reality. In this work, we show how the analysis of conflicting narratives, i.e. conflicting interpretive lenses through which political re…