RepMatch: Quantifying Cross-Instance Similarities in Representation Space
Advances in dataset analysis techniques have enabled more sophisticated approaches to analyzing and characterizing training data instances, often categorizing data based on attributes such as ``difficulty''. In this work, we introduce RepMatch, a novel method that characterizes data through the lens of similarity. RepMatch quantifies the similarity between subsets of training instances by comparing the knowledge encoded in models trained on them, overcoming the limitations of existing analysis methods that focus solely on individual instances and are restricted to within-dataset analysis. Our framework allows for a broader evaluation, enabling similarity comparisons across arbitrary subsets of instances, supporting both dataset-to-dataset and instance-to-dataset analyses. We validate the effectiveness of RepMatch across multiple NLP tasks, datasets, and models. Through extensive experimentation, we demonstrate that RepMatch can effectively compare datasets, identify more representative subsets of a dataset (that lead to better performance than randomly selected subsets of equivalent size), and uncover heuristics underlying the construction of some challenge datasets.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders
The Universality Hypothesis in large language models (LLMs) claims that different models converge towards similar concept representations in their latent spaces. Providing evidence for this hypothesis would enable resear…
Dictionary LearningTowards Predicting the Success of Transfer-based Attacks by Quantifying Shared Feature Representations
Much effort has been made to explain and improve the success of transfer-based attacks (TBA) on black-box computer vision models. This work provides the first attempt at a priori prediction of attack success by identifyi…
Multiple Instance Learning with Bag Dissimilarities
Multiple instance learning (MIL) is concerned with learning from sets (bags) of objects (instances), where the individual instance labels are ambiguous. In this setting, supervised learning cannot be applied directly. Of…
Multiple Instance LearningAudio-Visual Instance Discrimination with Cross-Modal Agreement
We present a self-supervised learning approach to learn audio-visual representations from video and audio. Our method uses contrastive learning for cross-modal discrimination of video from audio and vice-versa. We show t…
Action RecognitionAudio ClassificationContrastive LearningSelf-Supervised Action Recognition+2Quantifying Spatial Homogeneity of Urban Road Networks via Graph Neural Networks
Quantifying the topological similarities of different parts of urban road networks (URNs) enables us to understand the urban growth patterns. While conventional statistics provide useful information about characteristics…