DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales
Extracting temporal relationships over a range of scales is a hallmark of human perception and cognition -- and thus it is a critical feature of machine learning applied to real-world problems. Neural networks are either plagued by the exploding/vanishing gradient problem in recurrent neural networks (RNNs) or must adjust their parameters to learn the relevant time scales (e.g., in LSTMs). This paper introduces DeepSITH, a network comprising biologically-inspired Scale-Invariant Temporal History (SITH) modules in series with dense connections between layers. SITH modules respond to their inputs with a geometrically-spaced set of time constants, enabling the DeepSITH network to learn problems along a continuum of time-scales. We compare DeepSITH to LSTMs and other recent RNNs on several time series prediction and decoding tasks. DeepSITH achieves state-of-the-art performance on these problems.
Code (1)
Tasks
Time SeriesTime Series AnalysisTime Series PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Affordance-First Decomposition for Continual Learning in Video-Language Understanding
Continual learning for video--language understanding is increasingly important as models face non-stationary data, domains, and query styles, yet prevailing solutions blur what should stay stable versus what should adapt…
Continual LearningHow Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?
Tensor decompositions have been successfully applied to compress neural networks. The compression algorithms using tensor decompositions commonly minimize the approximation error on the weights. Recent work assumes the a…
Neural Network CompressionTensor DecompositionFast and Accurate Dual-Way Streaming PARAFAC2 for Irregular Tensors -- Algorithm and Application
How can we efficiently and accurately analyze an irregular tensor in a dual-way streaming setting where the sizes of two dimensions of the tensor increase over time? What types of anomalies are there in the dual-way stre…
Profit and loss decomposition in continuous time and approximations
Financial institutions and insurance companies that analyze the evolution and sources of profits and losses often look at risk factors only at discrete reporting dates, ignoring the detailed paths. Continuous-time decomp…
Context, Reasoning, and Hierarchy: A Cost-Performance Study of Compound LLM Agent Design in an Adversarial POMDP
Deploying compound LLM agents in adversarial, partially observable sequential environments requires navigating several design dimensions: (1) what the agent sees, (2) how it reasons, and (3) how tasks are decomposed acro…