Beyond receptive fields: sequence-pooled normalization can supply most of a sequence labeler's context
A convolutional sequence labeler's receptive field is routinely treated as the extent of the model's usable context: it sets dilation schedules, bounds streaming horizons, and underwrites locality claims. However, we show that this can be false: when a normalization layer computes statistics from the current input along the sequence at inference, those statistics open a sequence-spanning path that bypasses the convolutional receptive field to provide global context. We derive this from the layer's Jacobian (the criterion needs no experiment), and what the path carries has a closed form. On a synthetic labeling process with computable optima, the global summary that a sequence-spanning normalization encodes already supplies almost all of what a larger receptive field would buy where labels come in long runs: a network reaching 9 positions comes within 0.009 of the whole-sequence optimum, against a near-chance bound for its reach. Closing the path, by taking the same statistics per position, multiplies what enlarging the receptive field is worth by up to an order of magnitude on simulated genomes at every difficulty level tested and on real 1000 Genomes haplotypes. The same path also confounds attribution: ablating a trained network's receptive-field-enlarging blocks severs part of the path, overstating their contribution 8.3-16.1-fold relative to retraining from scratch. The substitution of normalization for receptive field fades as labels switch more often. Where labels run long, neither the receptive-field justification nor the ablation is wrong about its numbers, but both credit the wrong component.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulations
Learning representations that accurately capture long-range dependencies in sequential inputs -- including text, audio, and genomic data -- is a key problem in deep learning. Feed-forward convolutional models capture onl…
Audio Super-ResolutionSuper-Resolutiontext-classificationText ClassificationTemporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulations.
Learning representations that accurately capture long-range dependencies in sequential inputs --- including text, audio, and genomic data --- is a key problem in deep learning. Feed-forward convolutional models capture o…
Audio Super-ResolutionSuper-Resolutiontext-classificationText Classification3D human pose estimation with adaptive receptive fields and dilated temporal convolutions
In this work, we demonstrate that receptive fields in 3D pose estimation can be effectively specified using optical flow. We introduce adaptive receptive fields, a simple and effective method to aid receptive field selec…
3D Human Pose Estimation3D Pose EstimationOptical Flow EstimationPose Estimation+1Improved Training of Sparse Coding Variational Autoencoder via Weight Normalization
Learning a generative model of visual information with sparse and compositional features has been a challenge for both theoretical neuroscience and machine learning communities. Sparse coding models have achieved great s…
DecoderRetinotopic Mechanics derived using classical physics
The concept of a cell$'$s receptive field is a bedrock in systems neuroscience, and the classical static description of the receptive field has had enormous success in explaining the fundamental mechanisms underlying vis…
Sensitivity