Meaningful Pose-Based Sign Language Evaluation
We present a comprehensive study on meaningfully evaluating sign language utterances in the form of human skeletal poses. The study covers keypoint distance-based, embedding-based, and back-translation-based metrics. We show tradeoffs between different metrics in different scenarios through automatic meta-evaluation of sign-level retrieval and a human correlation study of text-to-pose translation across different sign languages. Our findings and the open-source pose-evaluation toolkit provide a practical and reproducible way of developing and evaluating sign language translation or generation systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Sign Language TranslationSimilar Papers 제목 키워드 기반
SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation
This article introduces semantically meaningful causal language modeling (SMCLM), a selfsupervised method of training autoregressive models to generate semantically equivalent text. Our approach involves using semantical…
Paraphrase GenerationA Benchmark for Early-stage Parkinson's Disease Detection from Speech
Early-stage Parkinson's disease (EarlyPD) detection from speech is clinically meaningful yet underexplored, and published results are hard to compare because studies differ in datasets, languages, tasks, evaluation proto…
Consistent Human Evaluation of Machine Translation across Language Pairs
Obtaining meaningful quality scores for machine translation systems through human evaluation remains a challenge given the high variability between human evaluators, partly due to subjective expectations for translation …
Machine TranslationTranslationMethodical Evaluation of Arabic Word Embeddings
Many unsupervised learning techniques have been proposed to obtain meaningful representations of words from text. In this study, we evaluate these various techniques when used to generate Arabic word embeddings. We first…
Document ClassificationLearning Word EmbeddingsNamed Entity Recognition (NER)Word EmbeddingsAre Large Language Models Fit For Guided Reading?
This paper looks at the ability of large language models to participate in educational guided reading. We specifically, evaluate their ability to generate meaningful questions from the input text, generate diverse questi…