Complexity for deep neural networks and other characteristics of deep feature representations
We define a notion of complexity, which quantifies the nonlinearity of the computation of a neural network, as well as a complementary measure of the effective dimension of feature representations. We investigate these observables both for trained networks for various datasets as well as explore their dynamics during training, uncovering in particular power law scaling. These observables can be understood in a dual way as uncovering hidden internal structure of the datasets themselves as a function of scale or depth. The entropic character of the proposed notion of complexity should allow to transfer modes of analysis from neuroscience and statistical physics to the domain of artificial neural networks. The introduced observables can be applied without any change to the analysis of biological neuronal systems.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Graph Input Representations for Machine Learning Applications in Urban Network Analysis
Understanding and learning the characteristics of network paths has been of particular interest for decades and has led to several successful applications. Such analysis becomes challenging for urban networks as their si…
BIG-bench Machine LearningInterpreting Speaker Characteristics in the Dimensions of Self-Supervised Speech Features
How do speech models trained through self-supervised learning structure their representations? Previous studies have looked at how information is encoded in feature vectors across different layers. But few studies have c…
Self-Supervised LearningUsing Clinical Drug Representations for Improving Mortality and Length of Stay Predictions
Drug representations have played an important role in cheminformatics. However, in the healthcare domain, drug representations have been underused relative to the rest of Electronic Health Record (EHR) data, due to the c…
Mortality PredictionTime SeriesTime Series AnalysisDD-CNN: Depthwise Disout Convolutional Neural Network for Low-complexity Acoustic Scene Classification
This paper presents a Depthwise Disout Convolutional Neural Network (DD-CNN) for the detection and classification of urban acoustic scenes. Specifically, we use log-mel as feature representations of acoustic signals for …
Acoustic Scene ClassificationClassificationGeneral ClassificationScene ClassificationCharacter-Aware Neural Networks for Arabic Named Entity Recognition for Social Media
Named Entity Recognition (NER) is the task of classifying or labelling atomic elements in the text into categories such as Person, Location or Organisation. For Arabic language, recognizing named entities is a challengin…
Feature EngineeringInformation RetrievalMachine Translationnamed-entity-recognition+6