Spline-based Transformers

2025-04-03 · Prashanth Chandran, Agon Serifi, Markus Gross, Moritz Bächer

We introduce Spline-based Transformers, a novel class of Transformer models that eliminate the need for positional encoding. Inspired by workflows using splines in computer animation, our Spline-based Transformers embed an input sequence of elements as a smooth trajectory in latent space. Overcoming drawbacks of positional encoding such as sequence length extrapolation, Spline-based Transformers also provide a novel way for users to interact with transformer latent spaces by directly manipulating the latent control points to create new latent trajectories and sequences. We demonstrate the superior performance of our approach in comparison to conventional positional encoding on a variety of datasets, ranging from synthetic 2D to large-scale real-world datasets of images, 3D shapes, and animations.

📄 PDF Abstract BibTeX arXiv:2504.02797

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Residual Connection 설명 없음

Adam 설명 없음

Attention 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

BSAT: B-Spline Adaptive Tokenizer for Long-Term Time Series Forecasting

2026-01-02 · Maximilian Reinwardt, Michael Eichelbeck, Matthias Althoff arxiv

Long-term time series forecasting using transformers is hampered by the quadratic complexity of self-attention and the rigidity of uniform patching, which may be misaligned with the data's semantic structure. In this pap…

Time Series Forecasting

Hyb-KAN ViT: Hybrid Kolmogorov-Arnold Networks Augmented Vision Transformer

2025-05-07 · Sainath Dey, Mitul Goswami, Jashika Sethi, Prasant Kumar Pattnaik

This study addresses the inherent limitations of Multi-Layer Perceptrons (MLPs) in Vision Transformers (ViTs) by introducing Hybrid Kolmogorov-Arnold Network (KAN)-ViT (Hyb-KAN ViT), a novel framework that integrates wav…

Edge Detection Instance Segmentation Kolmogorov-Arnold Networks object-detection+3

Kinematic Tokenization: Optimization-Based Continuous-Time Tokens for Learnable Decision Policies in Noisy Time Series

2026-01-15 · Griffin Kearney arxiv

Transformers are designed for discrete tokens, yet many real-world signals are continuous processes observed through noisy sampling. Discrete tokenizations (raw values, patches, finite differences) can be brittle in low …

Kolmogorov-Arnold Transformer

2024-09-16 · Xingyi Yang, Xinchao Wang

Transformers stand as the cornerstone of mordern deep learning. Traditionally, these models rely on multi-layer perceptron (MLP) layers to mix the information between channels. In this paper, we introduce the Kolmogorov-…

Image Classification

Superposition in Transformers: A Novel Way of Building Mixture of Experts

2024-12-31 · Ayoub Ben Chaliah, Hela Dellagi

Catastrophic forgetting remains a major challenge when adapting large language models (LLMs) to new tasks or domains. Conventional fine-tuning often overwrites existing knowledge, causing performance degradation on origi…

Mixture-of-Experts