AdaStop: Cost-Aware Early Stopping for DNN Test Selection
Existing methods for testing deep neural networks (DNNs) primarily prioritize test inputs likely to reveal model faults under a fixed labeling budget. In practice, choosing that budget is difficult: too little testing misses failures, while too much incurs unnecessary labeling costs. This work studies the stopping problem in DNN testing. We formulate testing as a cost--benefit decision process in which labeling an input incurs cost $c$ and discovering a fault yields value $v$. Based on this formulation, we introduce \textit{AdaStop}, a framework that estimates the marginal fault discovery rate during testing and stops labeling when the estimated rate falls below the threshold $τ= c/v$. Experiments across multiple datasets, architectures, and selection strategies show that $65$--$84\%$ of faults can be discovered using only $9$--$31\%$ of the labeling budget.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
AdaStop: adaptive statistical testing for sound comparisons of Deep RL agents
Recently, the scientific community has questioned the statistical reproducibility of many empirical results, especially in the field of machine learning. To contribute to the resolution of this reproducibility crisis, we…
Deep Reinforcement LearningMuJoCoReinforcement Learning (RL)ACE: Adaptive Constraint-aware Early Stopping in Hyperparameter Optimization
Deploying machine learning models requires high model quality and needs to comply with application constraints. That motivates hyperparameter optimization (HPO) to tune model configurations under deployment constraints. …
FairnessHyperparameter OptimizationDynamic Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling
Self-Consistency (SC) is a widely used method to mitigate hallucinations in Large Language Models (LLMs) by sampling the LLM multiple times and outputting the most frequent solution. Despite its benefits, SC results in s…
Statistical Early Stopping for Reasoning Models
While LLMs have seen substantial improvement in reasoning capabilities, they also sometimes overthink, generating unnecessary reasoning steps, particularly under uncertainty, given ill-posed or ambiguous queries. We intr…
FLOP-Efficient Training: Early Stopping Based on Test-Time Compute Awareness
Scaling training compute, measured in FLOPs, has long been shown to improve the accuracy of large language models, yet training remains resource-intensive. Prior work shows that increasing test-time compute (TTC)-for exa…