Stability of In-Context Learning: A Spectral Coverage Perspective
In-context learning (ICL) is a pivotal capability for the practical deployment of large-scale language models, yet its reliability can vary substantially with the number of demonstrations provided in the prompt. A central obstacle is that the target notion, \emph{distributional stability under demonstration resampling}, is expensive to measure directly at scale, making prompt-length selection largely heuristic. We therefore study a \emph{computable sufficient condition} based on a spectral-coverage proxy: the lower tail of the spectrum of a regularized empirical second-moment matrix formed from demonstration representations. Under sub-Gaussian representation assumptions, we derive a non-asymptotic sample-size requirement (a lower bound on $K$) that guarantees this proxy event with prescribed failure probability, yielding a conservative prompt-length recommendation produced by an observable two-stage estimator. In large-scale experiments, the resulting estimates consistently upper-bound empirical accuracy knee-points, which we treat only as a practical surrogate for the prompt-length transition rather than a definition of stability. On a smaller held-out subset, direct resampling-based distributional stability measurements further validate the intended stability interpretation. Finally, a validation-only calibration step tightens the conservatism (typically to about $1.03$--$1.20\times$) while preserving conservative ordering, providing practical and verifiable guidance for ICL prompt design.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Training-Conditional Coverage Bounds for Uniformly Stable Learning Algorithms
The training-conditional coverage performance of the conformal prediction is known to be empirically sound. Recently, there have been efforts to support this observation with theoretical guarantees. The training-conditio…
Conformal PredictionPredictionStability-Certified Reinforcement Learning via Spectral Normalization
In this article, two types of methods from different perspectives based on spectral normalization are described for ensuring the stability of the system controlled by a neural network. The first one is that the L2 gain o…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Machine learning for prediction of extreme statistics in modulation instability
A central area of research in nonlinear science is the study of instabilities that drive the emergence of extreme events. Unfortunately, experimental techniques for measuring such phenomena often provide only partial cha…
BIG-bench Machine LearningSpectral Stability of Pseudoinverse-Based Extreme Learning Machine
Extreme Learning Machine (ELM) computes output weights analytically using the Moore-Penrose pseudoinverse. Although this leads to fast training, its numerical stability depends strongly on the conditioning of the hidden …
Graph Learning Should Move Beyond Restrictive Views of Spectral and Message-Passing GNNs
Graph neural networks (GNNs) are commonly divided into message-passing neural networks (MPNNs) and spectral GNNs, reflecting two largely separate research traditions in machine learning and signal processing. While MPNNs…
Graph Learning