MM-ALT: A Multimodal Automatic Lyric Transcription System
Automatic lyric transcription (ALT) is a nascent field of study attracting increasing interest from both the speech and music information retrieval communities, given its significant application potential. However, ALT with audio data alone is a notoriously difficult task due to instrumental accompaniment and musical constraints resulting in degradation of both the phonetic cues and the intelligibility of sung lyrics. To tackle this challenge, we propose the MultiModal Automatic Lyric Transcription system (MM-ALT), together with a new dataset, N20EM, which consists of audio recordings, videos of lip movements, and inertial measurement unit (IMU) data of an earbud worn by the performing singer. We first adapt the wav2vec 2.0 framework from automatic speech recognition (ASR) to the ALT task. We then propose a video-based ALT method and an IMU-based voice activity detection (VAD) method. In addition, we put forward the Residual Cross Attention (RCA) mechanism to fuse data from the three modalities (i.e., audio, video, and IMU). Experiments show the effectiveness of our proposed MM-ALT system, especially in terms of noise robustness. Project page is at https://n20em.github.io.
Code (1)
Tasks
Action DetectionActivity DetectionAutomatic Speech RecognitionAutomatic Speech Recognition (ASR)Information RetrievalMusic Information RetrievalRetrievalspeech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
Listening Like a Judge: A Music-Aware Framework for Automatic Singing Performance Evaluation
Automatic singing quality assessment (SQA) requires evaluating lyrical correctness and musical fidelity while handling expressive variations. However, existing systems largely rely on either acoustic cues or lyric transc…
VietLyrics: A Large-Scale Dataset and Models for Vietnamese Automatic Lyrics Transcription
Automatic Lyrics Transcription (ALT) for Vietnamese music presents unique challenges due to its tonal complexity and dialectal variations, but remains largely unexplored due to the lack of a dedicated dataset. Therefore,…
Automatic Lyrics TranscriptionLyrics Transcription for Humans: A Readability-Aware Benchmark
Writing down lyrics for human consumption involves not only accurately capturing word sequences, but also incorporating punctuation and formatting for clarity and to convey contextual information. This includes song stru…
Automatic Lyrics TranscriptionGenre-conditioned Acoustic Models for Automatic Lyrics Transcription of Polyphonic Music
Lyrics transcription of polyphonic music is challenging not only because the singing vocals are corrupted by the background music, but also because the background music and the singing style vary across music genres, suc…
Automatic Lyrics TranscriptionEnd-to-end lyrics Recognition with Voice to Singing Style Transfer
Automatic transcription of monophonic/polyphonic music is a challenging task due to the lack of availability of large amounts of transcribed data. In this paper, we propose a data augmentation method that converts natura…
Data AugmentationLanguage ModelingLanguage ModellingStyle Transfer+1