Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI
Understanding the relationship between tongue motion patterns during speech and their resulting speech acoustic outcomes -- i.e., articulatory-acoustic relation -- is of great importance in assessing speech quality and developing innovative treatment and rehabilitative strategies. This is especially important when evaluating and detecting abnormal articulatory features in patients with speech-related disorders. In this work, we aim to develop a framework for detecting speech motion anomalies in conjunction with their corresponding speech acoustics. This is achieved through the use of a deep cross-modal translator trained on data from healthy individuals only, which bridges the gap between 4D motion fields obtained from tagged MRI and 2D spectrograms derived from speech acoustic data. The trained translator is used as an anomaly detector, by measuring the spectrogram reconstruction quality on healthy individuals or patients. In particular, the cross-modal translator is likely to yield limited generalization capabilities on patient data, which includes unseen out-of-distribution patterns and demonstrates subpar performance, when compared with healthy individuals.~A one-class SVM is then used to distinguish the spectrograms of healthy individuals from those of patients. To validate our framework, we collected a total of 39 paired tagged MRI and speech waveforms, consisting of data from 36 healthy individuals and 3 tongue cancer patients. We used both 3D convolutional and transformer-based deep translation models, training them on the healthy training set and then applying them to both the healthy and patient testing sets. Our framework demonstrates a capability to detect abnormal patient data, thereby illustrating its potential in enhancing the understanding of the articulatory-acoustic relation for both healthy individuals and patients.
Code (0)
등록된 구현이 없습니다.
Tasks
Anomaly DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Uncertainty-Weighted Image-Event Multimodal Fusion for Video Anomaly Detection
Most existing video anomaly detectors rely solely on RGB frames, which lack the temporal resolution needed to capture abrupt or transient motion cues, key indicators of anomalous events. To address this limitation, we pr…
Anomaly DetectionAnomaly Detection In Surveillance VideosVideo Anomaly DetectionVideo UnderstandingMultimodal Motion Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection
Anomalies are rare and anomaly detection is often therefore framed as One-Class Classification (OCC), i.e. trained solely on normalcy. Leading OCC techniques constrain the latent representations of normal motions to limi…
2D Human Pose EstimationAnomaly DetectionHuman Pose ForecastingOne-Class Classification+1Event Stream based Multi-Modal Video Anomaly Detection: A Benchmark Dataset and Algorithms
Video anomaly detection (VAD) is critical for automated surveillance but remains fragile under challenging conditions such as illumination variations, fast motion, and complex backgrounds when relying solely on visible l…
Video Anomaly DetectionJust Dance with $π$! A Poly-modal Inductor for Weakly-supervised Video Anomaly Detection
Weakly-supervised methods for video anomaly detection (VAD) are conventionally based merely on RGB spatio-temporal features, which continues to limit their reliability in real-world scenarios. This is due to the fact tha…
Anomaly DetectionOptical Flow EstimationVideo Anomaly DetectionWeakly-supervised Video Anomaly DetectionJust Dance with pi! A Poly-modal Inductor for Weakly-supervised Video Anomaly Detection
Weakly-supervised methods for video anomaly detection (VAD) are conventionally based merely on RGB spatio-temporal features, which continues to limit their reliability in real-world scenarios. This is due to the fact…
Anomaly DetectionOptical Flow EstimationVideo Anomaly DetectionWeakly-supervised Video Anomaly Detection