Contamination Report for Multilingual Benchmarks

2024-10-21 · Sanchit Ahuja, Varun Gumma, Sunayana Sitaram

Benchmark contamination refers to the presence of test datasets in Large Language Model (LLM) pre-training or post-training data. Contamination can lead to inflated scores on benchmarks, compromising evaluation results and making it difficult to determine the capabilities of models. In this work, we study the contamination of popular multilingual benchmarks in LLMs that support multiple languages. We use the Black Box test to determine whether $7$ frequently used multilingual benchmarks are contaminated in $7$ popular open and closed LLMs and find that almost all models show signs of being contaminated with almost all the benchmarks we test. Our findings can help the community determine the best set of benchmarks to use for multilingual evaluation.

📄 PDF Abstract BibTeX arXiv:2410.16186

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage ModellingLarge Language Model

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Obscuring Data Contamination Through Translation: Evidence from Arabic Corpora

2026-01-21 · Chaymaa Abbas, Nour Shamaa, Mariette Awad arxiv

Data contamination undermines the validity of Large Language Model evaluation by enabling models to rely on memorized benchmark content rather than true generalization. While prior work has proposed contamination detecti…

An Open Source Data Contamination Report for Large Language Models

2023-10-26 · Yucheng Li, Frank Guerin, Chenghua Lin

Data contamination in model evaluation has become increasingly prevalent with the growing popularity of large language models. It allows models to "cheat" via memorisation instead of displaying true capabilities. Therefo…

HellaSwag Language Modeling Language Modelling Large Language Model+2

Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness

2026-08-10 · Pinzhen Chen, Koel Dutta Chowdhury, Xiaoya Xu, David Tan 외 hf

Multilingual translation benchmarks are typically sourced in English and translated into other languages, treating language pairs as the unit of evaluation---a design that is prone to contamination over time and overlook…

Data Contamination Can Cross Language Barriers

2024-06-19 · Feng Yao, Yufan Zhuang, Zihao Sun, Sunan Xu 외

The opacity in developing large language models (LLMs) is raising growing concerns about the potential contamination of public benchmarks in the pre-training data. Existing contamination detection methods are typically b…

Memorization

Translation as a Scalable Proxy for Multilingual Evaluation

2026-01-16 · Sheriff Issaka, Erick Rosas Gonzalez, Lieqi Liu, Evans Kofi Agyei 외 arxiv

The rapid proliferation of LLMs has created a critical evaluation paradox: while LLMs claim multilingual proficiency, comprehensive non-machine-translated benchmarks exist for fewer than 30 languages, leaving >98% of the…