How Out-of-Distribution Detection Learning Theory Enhances Transformer: Learnability and Reliability
Transformers excel in natural language processing and computer vision tasks. However, they still face challenges in generalizing to Out-of-Distribution (OOD) datasets, i.e. data whose distribution differs from that seen during training. OOD detection aims to distinguish outliers while preserving in-distribution (ID) data performance. This paper introduces the OOD detection Probably Approximately Correct (PAC) Theory for transformers, which establishes the conditions for data distribution and model configurations for the OOD detection learnability of transformers. It shows that outliers can be accurately represented and distinguished with sufficient data under conditions. The theoretical implications highlight the trade-off between theoretical principles and practical training paradigms. By examining this trade-off, we naturally derived the rationale for leveraging auxiliary outliers to enhance OOD detection. Our theory suggests that by penalizing the misclassification of outliers within the loss function and strategically generating soft synthetic outliers, one can robustly bolster the reliability of transformer networks. This approach yields a novel algorithm that ensures learnability and refines the decision boundaries between inliers and outliers. In practice, the algorithm consistently achieves state-of-the-art (SOTA) performance across various data formats.
Code (0)
등록된 구현이 없습니다.
Tasks
Image ClassificationLearning TheoryOut-of-Distribution DetectionSimilar Papers 제목 키워드 기반
On the Learnability of Out-of-distribution Detection
Supervised learning aims to train a classifier under the assumption that training and test data are from the same distribution. To ease the above assumption, researchers have studied a more realistic setting: out-of-dist…
Learning TheoryOut-of-Distribution DetectionOut of Distribution (OOD) DetectionPAC learningA Closer Look at the Learnability of Out-of-Distribution (OOD) Detection
Machine learning algorithms often encounter different or "out-of-distribution" (OOD) data at deployment time, and OOD detection is frequently employed to detect these examples. While it works reasonably well in practice,…
Learning TheoryOut of Distribution (OOD) DetectionPAC learningIs Out-of-Distribution Detection Learnable?
Supervised learning aims to train a classifier under the assumption that training and test data are from the same distribution. To ease the above assumption, researchers have studied a more realistic setting: out-of-dist…
DiversityLearning TheoryOut-of-Distribution DetectionOut of Distribution (OOD) Detection+1What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages
What can large language models learn? By definition, language models (LM) are distributions over strings. Therefore, an intuitive way of addressing the above question is to formalize it as a matter of learnability of cla…
Language ModelingLanguage ModellingCharacterizing Online and Private Learnability under Distributional Constraints via Generalized Smoothness
Understanding minimal assumptions that enable learning and generalization is perhaps the central question of learning theory. Several celebrated results in statistical learning theory, such as the VC theorem and Littlest…
Decision Making