On Leakage in Some Popular Benchmarks on Graphs
A number of benchmarks are based on graphs. Edges are typically split into train, validation and test splits, using a random partition. Leakage has been discovered in a number of popular benchmarks; FB15k has been replaced by FB15k-237 and WN18 has been replaced by WN18RR, though leakage has been reported even after these corrections. This paper will report a new type of leakage, $A$-leakage, on benchmarks for synonym-antonym classification. $A$-leakage infers labels for pairs of words in the test split, $w_i , w_j$, by exploiting labels on paths from $w_i$ to $w_j$ in the training split. We conclude that it is safer to partition vertices, $V$, than edges, $E$.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Data Leakage in Visual Datasets
We analyze data leakage in visual datasets. Data leakage refers to images in evaluation benchmarks that have been seen during training, compromising fair model evaluation. Given that large-scale datasets are often source…
Image RetrievalLessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks
Large Language Models (LLMs) are widely utilized in software engineering (SE) tasks, such as code generation and automated program repair. However, their reliance on extensive and often undisclosed pre-training datasets …
Code GenerationProgram RepairAre Large Language Models Memorizing Bug Benchmarks?
Large Language Models (LLMs) have become integral to various software engineering tasks, including code generation, bug detection, and repair. To evaluate model performance in these domains, numerous bug benchmarks conta…
Code GenerationMemorizationA Graph Symmetrisation Bound on Channel Information Leakage under Blowfish Privacy
Blowfish privacy is a recent generalisation of differential privacy that enables improved utility while maintaining privacy policies with semantic guarantees, a factor that has driven the popularity of differential priva…
CenterYou: A cloud-based Approach to Simplify Android Privacy Management
With mobile applications and associated services becoming increasingly popular, concerns are being raised about private data leakages have raised. Previous solutions to this well-known set of problems have approached it …
Decision MakingManagement