A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation
Retrieval-Augmented Generation (RAG) systems are often evaluated using final answer accuracy, even though their failures can originate from preprocessing, retrieval, context packing, or generation. This paper presents a controlled empirical study of RAG sensitivity, robustness, and stability across 56 experimental runs. We evaluate how chunk size, retrieval depth (top k), embedding-based reranking, probabilistic retrieval noise, and repeated seeded runs affect retrieval, context packing, and generation behavior. Using a fixed 500-question QA subset mapped to 20,958 unique corpus contexts, we analyze both final answer metrics and intermediate failure modes. Across these experiments, retrieval-oriented metrics improved under broader retrieval settings, while downstream exact-match and F1 scores often behaved non-monotonically. We also observe preprocessing-induced answer loss under smaller chunk sizes, progressive degradation under retrieval corruption, and higher observed variance in broader retrieval regimes. These findings suggest that RAG evaluation should include sensitivity, robustness, stability, and multi-stage failure analysis rather than relying only on final answer accuracy.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Stability and Sensitivity Analysis of Multi-Vendor, Multi-Terminal HVDC Systems
The stability of multi-vendor, multi-terminal HVDC systems can be analyzed in frequency domain by black-box impedance models using the generalized Nyquist stability criterion. Based on the impedance stability analysis, a…
SensitivityOn the Stability of Neural Networks in Deep Learning
Deep learning has achieved remarkable success across a wide range of tasks, but its models often suffer from instability and vulnerability: small changes to the input may drastically affect predictions, while optimizatio…
Adversarial RobustnessGeneralized Passivity Sensitivity Methodology for Small-Signal Stability Analysis
This paper proposes a generalized passivity sensitivity analysis for power system stability studies. The method uncovers the most effective instability mitigation actions for both device-level and system-level investigat…
SensitivityRobustness of Agentic AI Systems via Adversarially-Aligned Jacobian Regularization
As Large Language Models (LLMs) transition into autonomous multi-agent ecosystems, robust minimax training becomes essential yet remains prone to instability when highly non-linear policies induce extreme local curvature…
Efficient Sampling and Sensitivity Analysis of Rare Transient Instability Events via Subset Simulation
Assessing the risk of low-probability high-impact transient instability (TI) events is crucial for ensuring robust and stable power system operation under high uncertainty. However, direct Monte Carlo (DMC) simulation fo…
Sensitivity