Isotropy Matters: Soft-ZCA Whitening of Embeddings for Semantic Code Search
Low isotropy in an embedding space impairs performance on tasks involving semantic inference. Our study investigates the impact of isotropy on semantic code search performance and explores post-processing techniques to mitigate this issue. We analyze various code language models, examine isotropy in their embedding spaces, and its influence on search effectiveness. We propose a modified ZCA whitening technique to control isotropy levels in embeddings. Our results demonstrate that Soft-ZCA whitening improves the performance of pre-trained code language models and can complement contrastive fine-tuning.
Code (1)
Tasks
Code SearchMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Whitening Sentence Representations for Better Semantics and Faster Retrieval
Pre-training models such as BERT have achieved great success in many natural language processing tasks. However, how to obtain better sentence representation through these pre-training models is still worthy to exploit. …
RetrievalSentenceAre ID Embeddings Necessary? Whitening Pre-trained Text Embeddings for Effective Sequential Recommendation
Recent sequential recommendation models have combined pre-trained text embeddings of items with item ID embeddings to achieve superior recommendation performance. Despite their effectiveness, the expressive power of text…
Sequential RecommendationExploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity
Previous work has shown that the representations output by contextual language models are more anisotropic than static type embeddings, and typically display outlier dimensions. This seems to be true for both monolingual…
Semantic SimilaritySemantic Textual SimilaritySentenceSentence SimilarityUSTCCTSU at SemEval-2024 Task 1: Reducing Anisotropy for Cross-lingual Semantic Textual Relatedness Task
Cross-lingual semantic textual relatedness task is an important research task that addresses challenges in cross-lingual communication and text understanding. It helps establish semantic connections between different lan…
Information RetrievalMachine TranslationRetrievalSentence+1Calibrated Similarity for Reliable Geometric Analysis of Embedding Spaces
While raw cosine similarity in pretrained embedding spaces exhibits strong rank correlation with human judgments, anisotropy induces systematic miscalibration of absolute values: scores concentrate in a narrow high-simil…