Sharp Rate of Convergence for Deep Neural Network Classifiers under the Teacher-Student Setting
Classifiers built with neural networks handle large-scale high dimensional data, such as facial images from computer vision, extremely well while traditional statistical methods often fail miserably. In this paper, we attempt to understand this empirical success in high dimensional classification by deriving the convergence rates of excess risk. In particular, a teacher-student framework is proposed that assumes the Bayes classifier to be expressed as ReLU neural networks. In this setup, we obtain a sharp rate of convergence, i.e., $\tilde{O}_d(n^{-2/3})$, for classifiers trained using either 0-1 loss or hinge loss. This rate can be further improved to $\tilde{O}_d(n^{-1})$ when the data distribution is separable. Here, $n$ denotes the sample size. An interesting observation is that the data dimension only contributes to the $\log(n)$ term in the above rates. This may provide one theoretical explanation for the empirical successes of deep neural networks in high dimensional classification, particularly for structured data.
Code (0)
등록된 구현이 없습니다.
Tasks
General ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Deep interpretable architecture for plant diseases classification
Recently, many works have been inspired by the success of deep learning in computer vision for plant diseases classification. Unfortunately, these end-to-end deep classifiers lack transparency which can limit their adopt…
ClassificationGeneral ClassificationEstimating a sharp convergence bound for randomized ensembles
When randomized ensembles such as bagging or random forests are used for binary classification, the prediction error of the ensemble tends to decrease and stabilize as the number of classifiers increases. However, the pr…
Binary ClassificationDensity EstimationPredictionResidual-as-Teacher: Mitigating Bias Propagation in Student--Teacher Estimation
We study statistical estimation in a student--teacher setting, where predictions from a pre-trained teacher are used to guide a student model. A standard approach is to train the student to directly match the teacher's o…
Classification with Deep Neural Networks and Logistic Loss
Deep neural networks (DNNs) trained with the logistic loss (i.e., the cross entropy loss) have made impressive advancements in various binary classification tasks. However, generalization analysis for binary classificati…
Binary ClassificationClassificationGeneralization BoundsSADT: Combining Sharpness-Aware Minimization with Self-Distillation for Improved Model Generalization
Methods for improving deep neural network training times and model generalizability consist of various data augmentation, regularization, and optimization approaches, which tend to be sensitive to hyperparameter settings…
Data Augmentation