Stochastic Optimization of Plain Convolutional Neural Networks with Simple methods

2020-01-24 · Yahia Assiri

Convolutional neural networks have been achieving the best possible accuracies in many visual pattern classification problems. However, due to the model capacity required to capture such representations, they are often oversensitive to overfitting and therefore require proper regularization to generalize well. In this paper, we present a combination of regularization techniques which work together to get better performance, we built plain CNNs, and then we used data augmentation, dropout and customized early stopping function, we tested and evaluated these techniques by applying models on five famous datasets, MNIST, CIFAR10, CIFAR100, SVHN, STL10, and we achieved three state-of-the-art-of (MNIST, SVHN, STL10) and very high-Accuracy on the other two datasets.

📄 PDF Abstract BibTeX arXiv:2001.08856

Code (1)

junaidaliop/MNIST-SOPCNN pytorch

Tasks

Data Augmentation Image Classification Stochastic Optimization

Methods 이 논문이 사용한 방법론

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Early Stopping Early Stopping is a regularization technique for deep neural networks that stops training when parameter updates no longer begin to yield improves on a validation set. In…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

A Unified Analysis of Stochastic Optimization Methods Using Jump System Theory and Quadratic Constraints

2017-06-25 · Bin Hu, Peter Seiler, Anders Rantzer

We develop a simple routine unifying the analysis of several important recently-developed stochastic optimization methods including SAGA, Finito, and stochastic dual coordinate ascent (SDCA). First, we show an intrinsic …

Stochastic Optimization

Towards Understanding the Generalization Bias of Two Layer Convolutional Linear Classifiers with Gradient Descent

2018-02-13 · Yifan Wu, Barnabas Poczos, Aarti Singh

A major challenge in understanding the generalization of deep learning is to explain why (stochastic) gradient descent can exploit the network architecture to find solutions that have good generalization performance when…

Can Implicit Bias Explain Generalization? Stochastic Convex Optimization as a Case Study

2020-03-13 · NeurIPS 2020 12 · Assaf Dauber, Meir Feder, Tomer Koren, Roi Livni

The notion of implicit bias, or implicit regularization, has been suggested as a means to explain the surprising generalization ability of modern-days overparameterized learning algorithms. This notion refers to the tend…

Convergence rates for the Adam optimizer

2024-07-29 · Steffen Dereich, Arnulf Jentzen

Stochastic gradient descent (SGD) optimization methods are nowadays the method of choice for the training of deep neural networks (DNNs) in artificial intelligence systems. In practically relevant training problems, usua…

Stochastic Optimization

Closing the Gap: Tighter Analysis of Alternating Stochastic Gradient Methods for Bilevel Problems

2021-12-01 · NeurIPS 2021 12 · Tianyi Chen, Yuejiao Sun, Wotao Yin

Stochastic nested optimization, including stochastic compositional, min-max, and bilevel optimization, is gaining popularity in many machine learning applications. While the three problems share a nested structure, exist…

Bilevel Optimization