Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews
Peer review plays a central role in the NLP publication process, but is susceptible to various biases. Here, we study language-of-study (LoS) bias: the tendency for reviewers to evaluate a paper differently based on the language(s) it studies, rather than its scientific merit. Despite being explicitly flagged in reviewing guidelines, such biases are poorly understood. Prior work treats such comments as part of broader categories of weak or unconstructive reviews without defining them as a distinct form of bias. We present the first systematic characterization of LoS bias, distinguishing negative and positive forms, and introduce the human-annotated dataset LOBSTER (Language-Of-study Bias in ScienTific pEer Review) and a method achieving 87.37 macro F1 for detection. We analyze 15,645 reviews to estimate how negative and positive biases differ with respect to the LoS, and find that non-English papers face substantially higher bias rates than English-only ones, with negative bias consistently outweighing positive bias. Finally, we identify four subcategories of negative bias, and find that demanding unjustified cross-lingual generalization is the most dominant form. We publicly release all resources to support work on fairer reviewing practices in NLP and beyond.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Multilingual Hidden Prompt Injection Attacks on LLM-Based Academic Reviewing
Large language models (LLMs) are increasingly considered for use in high-impact workflows, including academic peer review. However, LLMs are vulnerable to document-level hidden prompt injection attacks. In this work, we …
Is China Entering WTO or shijie maoyi zuzhi--a Corpus Study of English Acronyms in Chinese Newspapers
This is one of the first studies that quantitatively examine the usage of English acronyms (e.g. WTO) in Chinese texts. Using newspaper corpora, I try to answer 1) for all instances of a concept that has an English acron…
TranslationOpenMSD: Towards Multilingual Scientific Documents Similarity Measurement
We develop and evaluate multilingual scientific documents similarity measurement models in this work. Such models can be used to find related works in different languages, which can help multilingual researchers find and…
Contrastive Learning of English Language and Crystal Graphs for Multimodal Representation of Materials Knowledge
Artificial intelligence (AI) is increasingly used for the inverse design of materials, such as crystals and molecules. Existing AI research on molecules has integrated chemical structures of molecules with textual knowle…
Contrastive LearningZero-shot GeneralizationScoping Review of Active Learning Strategies and their Evaluation Environments for Entity Recognition Tasks
We conducted a scoping review for active learning in the domain of natural language processing (NLP), which we summarize in accordance with the PRISMA-ScR guidelines as follows: Objective: Identify active learning strate…
Active LearningArticles