FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings
This paper presents factorized linear projection (FLiP) models for understanding pretrained sentence embedding spaces. We train FLiP models to recover the lexical content from multilingual (LaBSE), multimodal (SONAR) and API-based (Gemini) sentence embedding spaces in several high- and mid-resource languages. We show that FLiP can recall more than 75% of lexical content from the embeddings, significantly outperforming existing non-factorized baselines. Using this as a diagnostic tool, we uncover the modality and language biases across the selected sentence encoders and provide practitioners with intrinsic insights about the encoders without relying on conventional downstream evaluation tasks. Our implementation is public https://github.com/BUTSpeechFIT/FLiP.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Image Pivoting for Learning Multilingual Multimodal Representations
In this paper we propose a model to learn multimodal multilingual representations for matching images and sentences in different languages, with the aim of advancing multilingual versions of image search and image unders…
Image DescriptionImage RetrievalSemantic Textual SimilarityMulti-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations
With the aim of promoting and understanding the multilingual version of image search, we leverage visual object detection and propose a model with diverse multi-head attention to learn grounded multilingual multimodal re…
DiversityImage Retrievalobject-detectionObject Detection+1Naturalistic Causal Probing for Morpho-Syntax
Probing has become a go-to methodology for interpreting and analyzing deep neural models in natural language processing. However, there is still a lack of understanding of the limitations and weaknesses of various types …
SentenceSAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation
We propose the SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation learning framework. Unlike previous works on speech representation learning, which learns multilingual context…
Representation LearningRetrievalSentenceSentence Embedding+6Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description
We present the results from the second shared task on multimodal machine translation and multilingual image description. Nine teams submitted 19 systems to two tasks. The multimodal translation task, in which the source …
Image DescriptionMachine TranslationMultimodal Machine TranslationSentence+1