Divisive Normalization: Justification and Effectiveness as Efficient Coding Transform
Divisive normalization (DN) has been advocated as an effective nonlinear {\em efficient coding} transform for natural sensory signals with applications in biology and engineering. In this work, we aim to establish a connection between the DN transform and the statistical properties of natural sensory signals. Our analysis is based on the use of multivariate {\em t} model to capture some important statistical properties of natural sensory signals. The multivariate {\em t} model justifies DN as an approximation to the transform that completely eliminates its statistical dependency. Furthermore, using the multivariate {\em t} model and measuring statistical dependency with multi-information, we can precisely quantify the statistical dependency that is reduced by the DN transform. We compare this with the actual performance of the DN transform in reducing statistical dependencies of natural sensory signals. Our theoretical analysis and quantitative evaluations confirm DN as an effective efficient coding transform for natural sensory signals. On the other hand, we also observe a previously unreported phenomenon that DN may increase statistical dependencies when the size of pooling is small.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Appropriate kernels for Divisive Normalization explained by Wilson-Cowan equations
The interaction between wavelet-like sensors in Divisive Normalization is classically described through Gaussian kernels that decay with spatial distance, angular distance and frequency distance. However, simultaneous ex…
Cortical Divisive Normalization from Wilson-Cowan Neural Dynamics
Divisive Normalization and the Wilson-Cowan equations are influential models of neural interaction and saturation [Carandini and Heeger Nat.Rev.Neurosci. 2012; Wilson and Cowan Kybernetik 1973]. However, they have not be…
RelationDivisive Feature Normalization Improves Image Recognition Performance in AlexNet
Local divisive normalization provides a phenomenological description of many nonlinear response properties of neurons across visual cortical areas. To gain insight into the utility of this operation, we studied the effec…
Neural Networks with Divisive normalization for image segmentation with application in cityscapes dataset
One of the key problems in computer vision is adaptation: models are too rigid to follow the variability of the inputs. The canonical computation that explains adaptation in sensory neuroscience is divisive normalization…
Image SegmentationSegmentationSemantic SegmentationNormalizing the Normalizers: Comparing and Extending Network Normalization Schemes
Normalization techniques have only recently begun to be exploited in supervised learning tasks. Batch normalization exploits mini-batch statistics to normalize the activations. This was shown to speed up training and res…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+1