Learnt Contrastive Concept Embeddings for Sign Recognition

2023-08-18 · Ryan Wong, Necati Cihan Camgoz, Richard Bowden

In natural language processing (NLP) of spoken languages, word embeddings have been shown to be a useful method to encode the meaning of words. Sign languages are visual languages, which require sign embeddings to capture the visual and linguistic semantics of sign. Unlike many common approaches to Sign Recognition, we focus on explicitly creating sign embeddings that bridge the gap between sign language and spoken language. We propose a learning framework to derive LCC (Learnt Contrastive Concept) embeddings for sign language, a weakly supervised contrastive approach to learning sign embeddings. We train a vocabulary of embeddings that are based on the linguistic labels for sign video. Additionally, we develop a conceptual similarity loss which is able to utilise word embeddings from NLP methods to create sign embeddings that have better sign language to spoken language correspondence. These learnt representations allow the model to automatically localise the sign in time. Our approach achieves state-of-the-art keypoint-based sign recognition performance on the WLASL and BOBSL datasets.

📄 PDF Abstract BibTeX arXiv:2308.09515

Code (0)

등록된 구현이 없습니다.

Tasks

Word Embeddings

Methods 이 논문이 사용한 방법론

LCC Please enter a description about the method here

Focus 설명 없음

Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss

2023-10-02 · Jaidev Gill, Vala Vakilian, Christos Thrampoulidis

Supervised-contrastive loss (SCL) is an alternative to cross-entropy (CE) for classification tasks that makes use of similarities in the embedding space to allow for richer representations. In this work, we propose metho…

On Leveraging Variational Graph Embeddings for Open World Compositional Zero-Shot Learning

2022-04-23 · Muhammad Umer Anwaar, Zhihui Pan, Martin Kleinsteuber

Humans are able to identify and categorize novel compositions of known concepts. The task in Compositional Zero-Shot learning (CZSL) is to learn composition of primitive concepts, i.e. objects and states, in such a way t…

Compositional Zero-Shot Learning Image Retrieval Metric Learning Retrieval+1

Perfect match: Improved cross-modal embeddings for audio-visual synchronisation

2018-09-21 · Soo-Whan Chung, Joon Son Chung, Hong-Goo Kang

This paper proposes a new strategy for learning powerful cross-modal embeddings for audio-to-video synchronization. Here, we set up the problem as one of cross-modal retrieval, where the objective is to find the most rel…

Binary Classification Cross-Modal Retrieval Retrieval speech-recognition+3

Towards Counterfactual Image Manipulation via CLIP

2022-07-06 · Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jiahui Zhang 외

Leveraging StyleGAN's expressivity and its disentangled latent codes, existing methods can achieve realistic editing of different visual attributes such as age and gender of facial images. An intriguing yet challenging p…

counterfactual Image Manipulation

Ontology-based n-ball Concept Embeddings Informing Few-shot Image Classification

2021-09-19 · Mirantha Jayathilaka, Tingting Mu, Uli Sattler

We propose a novel framework named ViOCE that integrates ontology-based background knowledge in the form of $n$-ball concept embeddings into a neural network based vision architecture. The approach consists of two compon…

Few-Shot Image Classification image-classification Image Classification