Higher Order Linear Transformer
Following up on the linear transformer part of the article from Katharopoulos et al., that takes this idea from Shen et al., the trick that produces a linear complexity for the attention mechanism is re-used and extended to a second-order approximation of the softmax normalization.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Transformers Generalize DeepSets and Can be Extended to Graphs & Hypergraphs
We present a generalization of Transformers to any-order permutation invariant data (sets, graphs, and hypergraphs). We begin by observing that Transformers generalize DeepSets, or first-order (set-input) permutation inv…
2kGraph RegressionTransformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphs
We present a generalization of Transformers to any-order permutation invariant data (sets, graphs, and hypergraphs). We begin by observing that Transformers generalize DeepSets, or first-order (set-input) permutation inv…
2kGraph RegressionHyperedge PredictionSet-to-Graph PredictionHow Well Can Transformers Emulate In-context Newton's Method?
Transformer-based models have demonstrated remarkable in-context learning capabilities, prompting extensive research into its underlying mechanisms. Recent studies have suggested that Transformers can implement first-ord…
In-Context LearningregressionHigher-Order Transformer Derivative Estimates for Explicit Pathwise Learning Guarantees
An inherent challenge in computing fully-explicit generalization bounds for transformers involves obtaining covering number estimates for the given transformer class $T$. Crude estimates rely on a uniform upper bound on …
Generalization BoundsHigher Order Transformers: Enhancing Stock Movement Prediction On Multimodal Time-Series Data
In this paper, we tackle the challenge of predicting stock movements in financial markets by introducing Higher Order Transformers, a novel architecture designed for processing multivariate time-series data. We extend th…
DecoderStock Market PredictionTensor DecompositionTime Series