Sign Language Recognition Analysis using Multimodal Data
Voice-controlled personal and home assistants (such as the Amazon Echo and Apple Siri) are becoming increasingly popular for a variety of applications. However, the benefits of these technologies are not readily accessible to Deaf or Hard-ofHearing (DHH) users. The objective of this study is to develop and evaluate a sign recognition system using multiple modalities that can be used by DHH signers to interact with voice-controlled devices. With the advancement of depth sensors, skeletal data is used for applications like video analysis and activity recognition. Despite having similarity with the well-studied human activity recognition, the use of 3D skeleton data in sign language recognition is rare. This is because unlike activity recognition, sign language is mostly dependent on hand shape pattern. In this work, we investigate the feasibility of using skeletal and RGB video data for sign language recognition using a combination of different deep learning architectures. We validate our results on a large-scale American Sign Language (ASL) dataset of 12 users and 13107 samples across 51 signs. It is named as GMUASL51. We collected the dataset over 6 months and it will be publicly released in the hope of spurring further machine learning research towards providing improved accessibility for digital assistants.
Code (0)
등록된 구현이 없습니다.
Tasks
Activity RecognitionHuman Activity RecognitionSign Language RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Multimodal Language Analysis with Recurrent Multistage Fusion
Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities. Comprehending multimodal language requires modeling n…
Emotion RecognitionMultimodal Sentiment AnalysisSentiment AnalysisMultimodal Emotion Recognition and Sentiment Analysis in Multi-Party Conversation Contexts
Emotion recognition and sentiment analysis are pivotal tasks in speech and language processing, particularly in real-world scenarios involving multi-party, conversational data. This paper presents a multimodal approach t…
Emotion RecognitionMultimodal Emotion RecognitionSentiment AnalysisR1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning
In this work, we present the first application of Reinforcement Learning with Verifiable Reward (RLVR) to an Omni-multimodal large language model in the context of emotion recognition, a task where both visual and audio …
Emotion RecognitionLanguage ModelingLanguage ModellingLarge Language Model+4Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
Multimodal language analysis often considers relationships between features based on text and those based on acoustical and visual properties. Text features typically outperform non-text features in sentiment analysis or…
Emotion RecognitionMultimodal Sentiment AnalysisSentiment AnalysisWord EmbeddingsA Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis
Understanding expressed sentiment and emotions are two crucial factors in human multimodal language. This paper describes a Transformer-based joint-encoding (TBJE) for the task of Emotion Recognition and Sentiment Analys…
Emotion RecognitionMultimodal Sentiment AnalysisSentiment Analysis