Multimodal Segmentation for Vocal Tract Modeling

2024-06-22 · Rishi Jain, Bohan Yu, Peter Wu, Tejas Prabhune, Gopala Anumanchipalli

Accurate modeling of the vocal tract is necessary to construct articulatory representations for interpretable speech processing and linguistics. However, vocal tract modeling is challenging because many internal articulators are occluded from external motion capture technologies. Real-time magnetic resonance imaging (RT-MRI) allows measuring precise movements of internal articulators during speech, but annotated datasets of MRI are limited in size due to time-consuming and computationally expensive labeling methods. We first present a deep labeling strategy for the RT-MRI video using a vision-only segmentation approach. We then introduce a multimodal algorithm using audio to improve segmentation of vocal articulators. Together, we set a new benchmark for vocal tract modeling in MRI video segmentation and use this to release labels for a 75-speaker RT-MRI dataset, increasing the amount of labeled public RT-MRI data of the vocal tract by over a factor of 9. The code and dataset labels can be found at \url{rishiraij.github.io/multimodal-mri-avatar/}.

📄 PDF Abstract BibTeX arXiv:2406.15754

Code (0)

등록된 구현이 없습니다.

Tasks

Segmentation Video Segmentation Video Semantic Segmentation

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI

2026-05-18 · Daiqi Liu, Lukas Mulzer, Md Hasan, Nyvenn de Castro 외 arxiv

Segmenting vocal tract articulators in real-time MRI (rtMRI) is a challenging dynamic image segmentation problem characterized by low contrast, rapid motion, and limited spatial resolution. However, while rtMRI acquisiti…

Image Segmentation

Multimodal Laryngoscopic Video Analysis for Assisted Diagnosis of Vocal Fold Paralysis

2024-09-05 · Yucong Zhang, Xin Zou, Jinshan Yang, Wenjun Chen 외

This paper presents the Multimodal Laryngoscopic Video Analyzing System (MLVAS), a novel system that leverages both audio and video data to automatically extract key video segments and metrics from raw laryngeal videostr…

Keyword Spotting Segmentation

VocSegMRI: Multimodal Learning for Precise Vocal Tract Segmentation in Real-time MRI

2025-09-17 · Daiqi Liu, Johannes Enk, Maureen Stone, Fangxu Xing 외 arxiv

Accurate segmentation of articulatory structures in real-time MRI (rtMRI) remains challenging, as existing methods rely primarily on visual cues and overlook complementary information from synchronized speech signals. We…

Contrastive Learning

Open-Source Manually Annotated Vocal Tract Database for Automatic Segmentation from 3D MRI Using Deep Learning: Benchmarking 2D and 3D Convolutional and Transformer Networks

2025-01-08 · Subin Erattakulangara, Karthika Kelat, Katie Burnham, Rachel Balbi 외

Accurate segmentation of the vocal tract from magnetic resonance imaging (MRI) data is essential for various voice and speech applications. Manual segmentation is time intensive and susceptible to errors. This study aime…

Benchmarking Deep Learning Segmentation

A Transformer Model for Segmentation, Classification, and Caller Identification of Marmoset Vocalization

2024-10-30 · Bin Wu, Shinnosuke Takamichi, Sakriani Sakti, Satoshi Nakamura

Marmoset, a highly vocalized primate, has become a popular animal model for studying social-communicative behavior and its underlying mechanism comparing with human infant linguistic developments. In the study of vocal c…