Context Matters: Self-Attention for Sign Language Recognition
This paper proposes an attentional network for the task of Continuous Sign Language Recognition. The proposed approach exploits co-independent streams of data to model the sign language modalities. These different channels of information can share a complex temporal structure between each other. For that reason, we apply attention to synchronize and help capture entangled dependencies between the different sign language components. Even though Sign Language is multi-channel, handshapes represent the central entities in sign interpretation. Seeing handshapes in their correct context defines the meaning of a sign. Taking that into account, we utilize the attention mechanism to efficiently aggregate the hand features with their appropriate spatio-temporal context for better sign recognition. We found that by doing so the model is able to identify the essential Sign Language components that revolve around the dominant hand and the face areas. We test our model on the benchmark dataset RWTH-PHOENIX-Weather 2014, yielding competitive results.
Code (2)
Tasks
Sign Language RecognitionSimilar Papers 제목 키워드 기반
Translating Natural Language to SQL using Pointer-Generator Networks and How Decoding Order Matters
Translating natural language to SQL queries for table-based question answering is a challenging problem and has received significant attention from the research community. In this work, we extend a pointer-generator and …
Question AnsweringSemantic ParsingTable-based Question AnsweringYour Attention Matters: to Improve Model Robustness to Noise and Spurious Correlations
Self-attention mechanisms are foundational to Transformer architectures, supporting their impressive success in a wide range of tasks. While there are many self-attention variants, their robustness to noise and spurious …
Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Whole slide image (WSI) analysis presents significant computational challenges due to the massive number of patches in gigapixel images. While transformer architectures excel at modeling long-range correlations through s…
Survival AnalysisHow Time Matters: Learning Time-Decay Attention for Contextual Spoken Language Understanding in Dialogues
Spoken language understanding (SLU) is an essential component in conversational systems. Most SLU components treats each utterance independently, and then the following components aggregate the multi-turn information in …
Dialogue State TrackingImage CaptioningMachine TranslationSentence+3Attention Basin: Why Contextual Position Matters in Large Language Models
The performance of Large Language Models (LLMs) is significantly sensitive to the contextual position of information in the input. To investigate the mechanism behind this positional bias, our extensive experiments revea…