Spooky Action at a Distance: Normalization Layers Enable Side-Channel Spatial Communication
This work shows that normalization layers can facilitate a surprising degree of communication across the spatial dimensions of an input tensor. We study a toy localization task with a convolutional architecture and show that normalization layers enable an iterative message passing procedure, allowing information aggregation from well outside the local receptive field. Our results suggest that normalization layers should be employed with caution in applications such as diffusion-based trajectory generation, where maintaining a spatially limited receptive field is crucial.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Spooky effect in optimal OSPA estimation and how GOSPA solves it
In this paper, we show the spooky effect at a distance that arises in optimal estimation of multiple targets with the optimal sub-pattern assignment (OSPA) metric. This effect refers to the fact that if we have several i…
An analysis on metric-driven multi-target sensor management: GOSPA versus OSPA
This paper presents an analysis on sensor management using a cost function based on a multi-target metric, in particular, the optimal subpattern-assignment (OSPA) metric, the unnormalised OSPA (UOSPA) metric and the gene…
ManagementSpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effects
Machine-learned force fields (ML-FFs) combine the accuracy of ab initio methods with the efficiency of conventional force fields. However, current ML-FFs typically ignore electronic degrees of freedom, such as the total …
Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
The success of deep neural networks is in part due to the use of normalization layers. Normalization layers like Batch Normalization, Layer Normalization and Weight Normalization are ubiquitous in practice, as they impro…
Learning TheoryDivisive Feature Normalization Improves Image Recognition Performance in AlexNet
Local divisive normalization provides a phenomenological description of many nonlinear response properties of neurons across visual cortical areas. To gain insight into the utility of this operation, we studied the effec…