Communication breakdown: On the low mutual intelligibility between human and neural captioning
We compare the 0-shot performance of a neural caption-based image retriever when given as input either human-produced captions or captions generated by a neural captioner. We conduct this comparison on the recently introduced ImageCoDe data-set (Krojer et al., 2022) which contains hard distractors nearly identical to the images to be retrieved. We find that the neural retriever has much higher performance when fed neural rather than human captions, despite the fact that the former, unlike the latter, were generated without awareness of the distractors that make the task hard. Even more remarkably, when the same neural captions are given to human subjects, their retrieval performance is almost at chance level. Our results thus add to the growing body of evidence that, even when the ``language'' of neural models resembles English, this superficial resemblance might be deeply misleading.
Code (1)
Tasks
RetrievalSimilar Papers 제목 키워드 기반
Are Mutually Intelligible Languages Easier to Translate?
Two languages are considered mutually intelligible if their native speakers can communicate with each other, while using their own mother tongue. How does the fact that humans perceive a language pair as mutually intelli…
TranslationMeasuring cross-language intelligibility between Romance languages with computational tools
We present an analysis of mutual intelligibility in related languages applied for languages in the Romance family. We introduce a novel computational metric for estimating intelligibility based on lexical similarity usin…
Semantic SimilarityDrawing with Strangers: Population Scaling Drives Zero-Shot Mutual Intelligibility in Emergent Sketching
Generalization in emergent communication has largely focused on novel inputs or linguistic structures, yet the capacity for agents to communicate with strangers from strictly disjoint communities remains relatively unexp…
A Computational Model for the Assessment of Mutual Intelligibility Among Closely Related Languages
Closely related languages show linguistic similarities that allow speakers of one language to understand speakers of another language without having actively learned it. Mutual intelligibility varies in degree and is typ…
The similarity and Mutual Intelligibility between Amharic and Tigrigna Varieties
The present study has examined the similarity and the mutual intelligibility between Amharic and Tigrigna using three tools namely Levenshtein distance, intelligibility test and questionnaires. The study has shown that b…