Untangling in Invariant Speech Recognition
Encouraged by the success of deep neural networks on a variety of visual tasks, much theoretical and experimental work has been aimed at understanding and interpreting how vision networks operate. Meanwhile, deep neural networks have also achieved impressive performance in audio processing applications, both as sub-components of larger systems and as complete end-to-end systems by themselves. Despite their empirical successes, comparatively little is understood about how these audio models accomplish these tasks. In this work, we employ a recently developed statistical mechanical theory that connects geometric properties of network representations and the separability of classes to probe how information is untangled within neural networks trained to recognize speech. We observe that speaker-specific nuisance variations are discarded by the network's hierarchy, whereas task-relevant properties such as words and phonemes are untangled in later layers. Higher level concepts such as parts-of-speech and context dependence also emerge in the later layers of the network. Finally, we find that the deep representations carry out significant temporal untangling by efficiently extracting task-relevant features at each time step of the computation. Taken together, these findings shed light on how deep auditory models process time dependent input signals to achieve invariant speech recognition, and show how different concepts emerge through the layers of the network.
Code (1)
Tasks
speech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
Probing emergent geometry in speech models via replica theory
The success of deep neural networks in visual tasks have motivated recent theoretical and empirical work to understand how these networks operate. Meanwhile, deep neural networks have also achieved impressive performance…
speech-recognitionSpeech RecognitionToward a Geometric Theory of Manifold Untangling
It has been hypothesized that the ventral stream processing for object recognition is based on a mechanism called cortically local subspace untangling. A mathematical abstraction of object recognition by the visual corte…
ObjectObject RecognitionAIPNet: Generative Adversarial Pre-training of Accent-invariant Networks for End-to-end Speech Recognition
As one of the major sources in speech variability, accents have posed a grand challenge to the robustness of speech recognition systems. In this paper, our goal is to build a unified end-to-end speech recognition system …
DecoderRepresentation Learningspeech-recognitionSpeech RecognitionUntangling Braids with Multi-agent Q-Learning
We use reinforcement learning to tackle the problem of untangling braids. We experiment with braids with 2 and 3 strands. Two competing players learn to tangle and untangle a braid. We interface the braid untangling prob…
OpenAI GymQ-Learningreinforcement-learningReinforcement Learning+1NIESR: Nuisance Invariant End-to-end Speech Recognition
Deep neural network models for speech recognition have achieved great success recently, but they can learn incorrect associations between the target and nuisance factors of speech (e.g., speaker identities, background no…
speech-recognitionSpeech Recognition