Exploring Strategies for Modeling Sign Language Phonology
Like speech, signs are composed of discrete, recombinable features called phonemes. Prior work shows that models which can recognize phonemes are better at sign recognition, motivating deeper exploration into strategies for modeling sign language phonemes. In this work, we learn graph convolution networks to recognize the sixteen phoneme "types" found in ASL-LEX 2.0. Specifically, we explore how learning strategies like multi-task and curriculum learning can leverage mutually useful information between phoneme types to facilitate better modeling of sign language phonemes. Results on the Sem-Lex Benchmark show that curriculum learning yields an average accuracy of 87% across all phoneme types, outperforming fine-tuning and multi-task strategies for most phoneme types.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
How Generative Spoken Language Modeling Encodes Noisy Speech: Investigation from Phonetics to Syntactics
We examine the speech modeling potential of generative spoken language modeling (GSLM), which involves using learned symbols derived from data rather than phonemes for speech analysis and synthesis. Since GSLM facilitate…
Language ModelingLanguage ModellingResynthesisModeling Markedness with a Split-and-Merger Model of Sound Change
The concept of {`}markedness{'} has been influential in phonology for almost a century. Theoretical phonology has found it useful to describe some segments as more {`}marked{'} than others, referring to a cluster of lang…
Speech vocoding for laboratory phonology
Using phonological speech vocoding, we propose a platform for exploring relations between phonology and speech processing, and in broader terms, for exploring relations between the abstract and physical structures of a s…
Speech Synthesistext-to-speechText to SpeechImprove Bilingual TTS Using Dynamic Language and Phonology Embedding
In most cases, bilingual TTS needs to handle three types of input scripts: first language only, second language only, and second language embedded in the first language. In the latter two situations, the pronunciation an…
Improving Sign Recognition with Phonology
We use insights from research on American Sign Language (ASL) phonology to train models for isolated sign language recognition (ISLR), a step towards automatic sign language understanding. Our key insight is to explicitl…
Sign Language Recognition