FragileFlow: Spectral Control of Correct-but-Fragile Predictions for Foundation Model Robustness
Robust adaptation of LLMs and VLMs is often evaluated by average accuracy or average consistency under perturbations. However, these averages can hide a structured failure mode: a prediction may remain correct while probability mass already flows from particular true classes toward systematic wrong competitors near the decision boundary. In this paper, we formalize this phenomenon as margin-aware error flow and introduce FragileFlow, a plug-in regularizer that uses a calibrated margin buffer to identify correct-but-fragile predictions and organize their off-class probability mass into a class-wise vulnerable-risk matrix. Theoretically, we provide the first PAC-Bayes upper bound for this margin-aware error-flow object, showing how empirical spectral control yields a conservative route to deterministic worst-class robustness under a stability condition. Experiments on multiple-choice LLM benchmarks and few-shot CLIP adaptation show that FragileFlow consistently improves the proposed theory-facing risk measures over matched baselines, yields perturbed worst-class accuracy gains in most settings, and preserves clean accuracy across comparisons.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Spectral-Spatial Synergistic Guided Network for Hyperspectral Salient Object Detection
Hyperspectral salient object detection aims to identify visually salient regions from hyperspectral images. Existing methods often fail because they fundamentally misunderstand the data, confusing incidental spectral var…
Computational EfficiencySalient Object DetectionGraph Neural Network Explanations are Fragile
Explainable Graph Neural Network (GNN) has emerged recently to foster the trust of using GNNs. Existing GNN explainers are developed from various perspectives to enhance the explanation performance. We take the first ste…
Adversarial AttackGraph Neural NetworkThree-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning
Federated parameter-efficient fine-tuning (PEFT) enables communication-efficient adaptation of large pretrained models on decentralized edge data, but it remains fragile under non-IID client heterogeneity. In low-rank ad…
parameter-efficient fine-tuningQuasi-Framelets: Robust Graph Neural Networks via Adaptive Framelet Convolution
This paper aims to provide a novel design of a multiscale framelet convolution for spectral graph neural networks (GNNs). While current spectral methods excel in various graph learning tasks, they often lack the flexibil…
Graph LearningGraph Neural NetworkNode ClassificationBreaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures
In schema-guided reasoning (SGR) pipelines, LLMs produce explicit intermediate structures -- rubrics, checklists, or verification queries -- before committing to a final decision. SGR is increasingly adopted because it p…