Small-footprint Deep Neural Networks with Highway Connections for Speech Recognition
For speech recognition, deep neural networks (DNNs) have significantly improved the recognition accuracy in most of benchmark datasets and application domains. However, compared to the conventional Gaussian mixture models, DNN-based acoustic models usually have much larger number of model parameters, making it challenging for their applications in resource constrained platforms, e.g., mobile devices. In this paper, we study the application of the recently proposed highway network to train small-footprint DNNs, which are {\it thinner} and {\it deeper}, and have significantly smaller number of model parameters compared to conventional DNNs. We investigated this approach on the AMI meeting speech transcription corpus which has around 70 hours of audio data. The highway neural networks constantly outperformed their plain DNN counterparts, and the number of model parameters can be reduced significantly without sacrificing the recognition accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
speech-recognitionSpeech RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Small-footprint Highway Deep Neural Networks for Speech Recognition
State-of-the-art speech recognition systems typically employ neural network acoustic models. However, compared to Gaussian mixture models, deep neural network (DNN) based acoustic models often have many more model parame…
speech-recognitionSpeech RecognitionKnowledge Distillation for Small-footprint Highway Networks
Deep learning has significantly advanced state-of-the-art of speech recognition in the past few years. However, compared to conventional Gaussian mixture acoustic models, neural network models are usually much larger, an…
Acoustic ModellingKnowledge Distillationspeech-recognitionSpeech RecognitionHighway Long Short-Term Memory RNNs for Distant Speech Recognition
In this paper, we extend the deep long short-term memory (DLSTM) recurrent neural networks by introducing gated direct connections between memory cells in adjacent layers. These direct links, called highway connections, …
Distant Speech Recognitionspeech-recognitionSpeech RecognitionInvestigating the effect of residual and highway connections in speech enhancement models
Residual and skip connections play an important role in many current generative models. Although their theoretical and numerical advantages are understood, their role in speech enhancement systems has not been in…
DenoisingSpeech DenoisingSpeech EnhancementEdgeSpeechNets: Highly Efficient Deep Neural Networks for Speech Recognition on the Edge
Despite showing state-of-the-art performance, deep learning for speech recognition remains challenging to deploy in on-device edge scenarios such as mobile and other consumer devices. Recently, there have been greater ef…
speech-recognitionSpeech Recognition