Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers

2021-02-09 · Shucong Zhang, Cong-Thanh Do, Rama Doddipatla, Erfan Loweimi, Peter Bell, Steve Renals

Although the lower layers of a deep neural network learn features which are transferable across datasets, these layers are not transferable within the same dataset. That is, in general, freezing the trained feature extractor (the lower layers) and retraining the classifier (the upper layers) on the same dataset leads to worse performance. In this paper, for the first time, we show that the frozen classifier is transferable within the same dataset. We develop a novel top-down training method which can be viewed as an algorithm for searching for high-quality classifiers. We tested this method on automatic speech recognition (ASR) tasks and language modelling tasks. The proposed method consistently improves recurrent neural network ASR models on Wall Street Journal, self-attention ASR models on Switchboard, and AWD-LSTM language models on WikiText-2.

📄 PDF Abstract BibTeX arXiv:2102.04697

Code (0)

등록된 구현이 없습니다.

Tasks

Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Language Modelling speech-recognition Speech Recognition

Methods 이 논문이 사용한 방법론

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

Variational Dropout Variational Dropout is a regularization technique based on dropout, but uses a variational inference grounded approach. In…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Weight Tying Weight Tying improves the performance of language models by tying (sharing) the weights of the embedding and softmax layers. This…

DropConnect DropConnect generalizes Dropout by randomly dropping the weights rather than the activations with probability $1-p$. DropConnect…

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

Activation Regularization Activation Regularization (AR), or $L\_{2}$ activation regularization, is regularization performed on activations as opposed to weights. It is usually used in conjunction with…

Joint Training of Cascaded CNN for Face Detection

2016-06-01 · CVPR 2016 6 · Hongwei Qin, Junjie Yan, Xiu Li, Xiaolin Hu

Cascade has been widely used in face detection, where classifier with low computation cost can be firstly used to shrink most of the background while keeping the recall. The cascade in detection is popularized by seminal…

Face Detection Region Proposal

Asymmetric Pruning for Learning Cascade Detectors

2013-03-25 · Sakrapee Paisitkriangkrai, Chunhua Shen, Anton Van Den Hengel

Cascade classifiers are one of the most important contributions to real-time object detection. Nonetheless, there are many challenging problems arising in training cascade detectors. One common issue is that the node cla…

object-detection Object Detection Real-Time Object Detection

CASCADE Your Datasets for Cross-Mode Knowledge Retrieval of Language Models

2025-04-02 · Runlong Zhou, Yi Zhang

Language models often struggle with cross-mode knowledge retrieval -- the ability to access knowledge learned in one format (mode) when queried in another. We demonstrate that models trained on multiple data sources (e.g…

Memorization Retrieval

Chained Cascade Network for Object Detection

2017-10-01 · ICCV 2017 10 · Wanli Ouyang, Kun Wang, Xin Zhu, Xiaogang Wang

Cascade is a widely used approach that rejects obvious negative samples at early stages for learning better classifier and faster inference. This paper presents chained cascade network (CC-Net). In this CC-Net, there are…

Object object-detection Object Detection

Cascaded Classifier for Pareto-Optimal Accuracy-Cost Trade-Off Using off-the-Shelf ANNs

2021-10-27 · Cecilia Latotzke, Johnson Loh, Tobias Gemmeke

Machine-learning classifiers provide high quality of service in classification tasks. Research now targets cost reduction measured in terms of average processing time or energy per solution. Revisiting the concept of cas…