DORA: Exploring Outlier Representations in Deep Neural Networks
Deep Neural Networks (DNNs) excel at learning complex abstractions within their internal representations. However, the concepts they learn remain opaque, a problem that becomes particularly acute when models unintentionally learn spurious correlations. In this work, we present DORA (Data-agnOstic Representation Analysis), the first data-agnostic framework for analyzing the representational space of DNNs. Central to our framework is the proposed Extreme-Activation (EA) distance measure, which assesses similarities between representations by analyzing their activation patterns on data points that cause the highest level of activation. As spurious correlations often manifest in features of data that are anomalous to the desired task, such as watermarks or artifacts, we demonstrate that internal representations capable of detecting such artifactual concepts can be found by analyzing relationships within neural representations. We validate the EA metric quantitatively, demonstrating its effectiveness both in controlled scenarios and real-world applications. Finally, we provide practical examples from popular Computer Vision models to illustrate that representations identified as outliers using the EA metric often correspond to undesired and spurious concepts.
Code (1)
Tasks
Decision MakingSimilar Papers 제목 키워드 기반
BiasDora: Exploring Hidden Biased Associations in Vision-Language Models
Existing works examining Vision-Language Models (VLMs) for social biases predominantly focus on a limited set of documented bias associations, such as gender:profession or race:crime. This narrow scope often overlooks a …
Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity
Previous work has shown that the representations output by contextual language models are more anisotropic than static type embeddings, and typically display outlier dimensions. This seems to be true for both monolingual…
Semantic SimilaritySemantic Textual SimilaritySentenceSentence SimilarityContextual Pandora's Box
Pandora's Box is a fundamental stochastic optimization problem, where the decision-maker must find a good alternative while minimizing the search cost of exploring the value of each alternative. In the original formulati…
Multi-Armed BanditsStochastic OptimizationExploring Efficient Learning of Small BERT Networks with LoRA and DoRA
While Large Language Models (LLMs) have revolutionized artificial intelligence, fine-tuning LLMs is extraordinarily computationally expensive, preventing smaller businesses and research teams with limited GPU resources f…
Sentiment AnalysisThe Effects of Quantum Randomness on a System Exhibiting Computational Creativity
We present experimental results on the effects of using quantum or 'truly' random numbers, as opposed to pseudorandom numbers, in a system that exhibits computational creativity (given its ability to compose original che…