Aspect-Guided Multi-Level Perturbation Analysis of Large Language Models in Automated Peer Review
We propose an aspect-guided, multi-level perturbation framework to evaluate the robustness of Large Language Models (LLMs) in automated peer review. Our framework explores perturbations in three key components of the peer review process-papers, reviews, and rebuttals-across several quality aspects, including contribution, soundness, presentation, tone, and completeness. By applying targeted perturbations and examining their effects on both LLM-as-Reviewer and LLM-as-Meta-Reviewer, we investigate how aspect-based manipulations, such as omitting methodological details from papers or altering reviewer conclusions, can introduce significant biases in the review process. We identify several potential vulnerabilities: review conclusions that recommend a strong reject may significantly influence meta-reviews, negative or misleading reviews may be wrongly interpreted as thorough, and incomplete or hostile rebuttals can unexpectedly lead to higher acceptance rates. Statistical tests show that these biases persist under various Chain-of-Thought prompting strategies, highlighting the lack of robust critical evaluation in current LLMs. Our framework offers a practical methodology for diagnosing these vulnerabilities, thereby contributing to the development of more reliable and robust automated reviewing systems.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Explanation-Guided Diagnosis of Machine Learning Evasion Attacks
Machine Learning (ML) models are susceptible to evasion attacks. Evasion accuracy is typically assessed using aggregate evasion rate, and it is an open question whether aggregate evasion rate enables feature-level diagno…
BIG-bench Machine LearningOpen-Ended Question AnsweringGTPC-SSCD: Gate-guided Two-level Perturbation Consistency-based Semi-Supervised Change Detection
Semi-supervised change detection (SSCD) utilizes partially labeled data and abundant unlabeled data to detect differences between multi-temporal remote sensing images. The mainstream SSCD methods based on consistency reg…
Change DetectionSemi-supervised Change DetectionExplainable Multimodal Aspect-Based Sentiment Analysis with Dependency-guided Large Language Model
Multimodal aspect-based sentiment analysis (MABSA) aims to identify aspect-level sentiments by jointly modeling textual and visual information, which is essential for fine-grained opinion understanding in social media. E…
Sentiment AnalysisASPECT: Node-Level Adaptive Spectral Fusion for Graph Contrastive Learning
Spectral graph contrastive learning often constructs low- and high-frequency views to capture complementary graph signals, but these views are commonly combined by graph-level or node-agnostic fusion rules. We show that …
Contrastive LearningDiscriminative Semantic Feature Pyramid Network with Guided Anchoring for Logo Detection
Recently, logo detection has received more and more attention for its wide applications in the multimedia field, such as intellectual property protection, product brand management, and logo duration monitoring. Unlike ge…
Managementobject-detectionObject Detection