Towards Annotation-Free Evaluation of Cross-Lingual Image Captioning

2020-12-09 · Aozhu Chen, Xinyi Huang, Hailan Lin, Xirong Li

Cross-lingual image captioning, with its ability to caption an unlabeled image in a target language other than English, is an emerging topic in the multimedia field. In order to save the precious human resource from re-writing reference sentences per target language, in this paper we make a brave attempt towards annotation-free evaluation of cross-lingual image captioning. Depending on whether we assume the availability of English references, two scenarios are investigated. For the first scenario with the references available, we propose two metrics, i.e., WMDRel and CLinRel. WMDRel measures the semantic relevance between a model-generated caption and machine translation of an English reference using their Word Mover's Distance. By projecting both captions into a deep visual feature space, CLinRel is a visual-oriented cross-lingual relevance measure. As for the second scenario, which has zero reference and is thus more challenging, we propose CMedRel to compute a cross-media relevance between the generated caption and the image content, in the same visual feature space as used by CLinRel. The promising results show high potential of the new metrics for evaluation with no need of references in the target language.

📄 PDF Abstract BibTeX arXiv:2012.04925

Code (0)

등록된 구현이 없습니다.

Tasks

Image Captioning Machine Translation Translation

ANHALTEN: Cross-Lingual Transfer for German Token-Level Reference-Free Hallucination Detection

2024-07-18 · Janek Herrlein, Chia-Chien Hung, Goran Glavaš

Research on token-level reference-free hallucination detection has predominantly focused on English, primarily due to the scarcity of robust datasets in other languages. This has hindered systematic investigations into t…

Cross-Lingual Transfer Hallucination

Multilingual Image Corpus: Annotation Protocol

2021-09-01 · RANLP 2021 9 · Svetla Koeva

In this paper, we present work in progress aimed at the development of a new image dataset with annotated objects. The Multilingual Image Corpus consists of an ontology of visual objects (based on WordNet) and a collecti…

Attribute image-classification Image Classification Image Retrieval+5

Multilingual Projection for Parsing Truly Low-Resource Languages

2016-01-01 · TACL 2016 1 · {\v{Z}}eljko Agi{\'c}, Anders Johannsen, Barbara Plank, H{\'e}ctor Mart{\'\i}nez Alonso 외

We propose a novel approach to cross-lingual part-of-speech tagging and dependency parsing for truly low-resource languages. Our annotation projection-based approach yields tagging and parsing models for over 100 languag…

Cross-Lingual Transfer Dependency Parsing Part-Of-Speech Tagging Transfer Learning

RECSA: Resource for Evaluating Cross-lingual Semantic Annotation

2014-05-01 · LREC 2014 5 · Achim Rettinger, Lei Zhang, Da{\v{s}}a Berovi{\'c}, Danijela Merkler 외

In recent years large repositories of structured knowledge (DBpedia, Freebase, YAGO) have become a valuable resource for language technologies, especially for the automatic aggregation of knowledge from textual data. One…

Articles Machine Translation

SpatialVOC2K: A Multilingual Dataset of Images with Annotations and Features for Spatial Relations between Objects

2018-11-01 · WS 2018 11 · Anja Belz, Adrian Muscat, Pierre Anguill, Mouhamadou Sow 외

We present SpatialVOC2K, the first multilingual image dataset with spatial relation annotations and object features for image-to-text generation, built using 2,026 images from the PASCAL VOC2008 dataset. The dataset inco…

Image to text Object Text Generation