Exploring Dimensionality Reduction Techniques in Multilingual Transformers
Both in scientific literature and in industry,, Semantic and context-aware Natural Language Processing-based solutions have been gaining importance in recent years. The possibilities and performance shown by these models when dealing with complex Language Understanding tasks is unquestionable, from conversational agents to the fight against disinformation in social networks. In addition, considerable attention is also being paid to developing multilingual models to tackle the language bottleneck. The growing need to provide more complex models implementing all these features has been accompanied by an increase in their size, without being conservative in the number of dimensions required. This paper aims to give a comprehensive account of the impact of a wide variety of dimensional reduction techniques on the performance of different state-of-the-art multilingual Siamese Transformers, including unsupervised dimensional reduction techniques such as linear and nonlinear feature extraction, feature selection, and manifold techniques. In order to evaluate the effects of these techniques, we considered the multilingual extended version of Semantic Textual Similarity Benchmark (mSTSb) and two different baseline approaches, one using the pre-trained version of several models and another using their fine-tuned STS version. The results evidence that it is possible to achieve an average reduction in the number of dimensions of $91.58\% \pm 2.59\%$ and $54.65\% \pm 32.20\%$, respectively. This work has also considered the consequences of dimensionality reduction for visualization purposes. The results of this study will significantly contribute to the understanding of how different tuning approaches affect performance on semantic-aware tasks and how dimensional reduction techniques deal with the high-dimensional embeddings computed for the STS task and their potential for highly demanding NLP tasks
Code (0)
등록된 구현이 없습니다.
Tasks
Dimensionality Reductionfeature selectionSemantic Textual SimilaritySTSSimilar Papers 제목 키워드 기반
A Review, Framework and R toolkit for Exploring, Evaluating, and Comparing Visualizations
This paper gives a review and synthesis of methods of evaluating dimensionality reduction techniques. Particular attention is paid to rank-order neighborhood evaluation metrics. A framework is created for exploring dimen…
Dimensionality ReductionExploring the Limits of KV Cache Compression in Visual Autoregressive Transformers
A fundamental challenge in Visual Autoregressive models is the substantial memory overhead required during inference to store previously generated representations. Despite various attempts to mitigate this issue through …
Dimensionality ReductionA Brief Survey on Representation Learning based Graph Dimensionality Reduction Techniques
Dimensionality reduction techniques map data represented on higher dimensions onto lower dimensions with varying degrees of information loss. Graph dimensionality reduction techniques adopt the same principle of providin…
Dimensionality ReductionGraph Representation LearningPhilosophyRepresentation Learning+1Going Beyond T-SNE: Exposing whatlies in Text Embeddings
We introduce whatlies, an open source toolkit for visually inspecting word and sentence embeddings. The project offers a unified and extensible API with current support for a range of popular embedding backends including…
Dimensionality ReductionSentenceSentence EmbeddingsWord EmbeddingsGoing Beyond T-SNE: Exposing \texttt{whatlies} in Text Embeddings
We introduce whatlies, an open source toolkit for visually inspecting word and sentence embeddings. The project offers a unified and extensible API with current support for a range of popular embedding backends including…
Dimensionality ReductionSentenceSentence EmbeddingsWord Embeddings