Extrapolation of Periodic Functions Using Binary Encoding of Continuous Numerical Values
We report the discovery that binary encoding allows neural networks to extrapolate periodic functions beyond their training bounds. We introduce Normalized Base-2 Encoding (NB2E) as a method for encoding continuous numerical values and demonstrate that, using this input encoding, vanilla multi-layer perceptrons (MLP) successfully extrapolate diverse periodic signals without prior knowledge of their functional form. Internal activation analysis reveals that NB2E induces bit-phase representations, enabling MLPs to learn and extrapolate signal structure independently of position.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Periodic Extrapolative Generalisation in Neural Networks
The learning of the simplest possible computational pattern -- periodicity -- is an open problem in the research of strong generalisation in neural networks. We formalise the problem of extrapolative generalisation for p…
BenchmarkingUltraViCo: Breaking Extrapolation Limits in Video Diffusion Transformers
Despite advances, video diffusion transformers still struggle to generalize beyond their training length, a challenge we term video length extrapolation. We identify two failure modes: model-specific periodic content rep…
Neural Networks Fail to Learn Periodic Functions and How to Fix It
Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally…
Inductive BiasMEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation
When the predicted sequence length exceeds the length seen during training, the transformer's inference accuracy diminishes. Existing relative position encoding methods, such as those based on the ALiBi technique, addres…
Self-Supervised Learning as Discrete Communication
Most self-supervised learning (SSL) methods learn continuous visual representations by aligning different views of the same input, offering limited control over how information is structured across representation dimensi…
Self-Supervised LearningImage Classification