Structure-Preserving Transformers for Sequences of SPD Matrices
In recent years, Transformer-based auto-attention mechanisms have been successfully applied to the analysis of a variety of context-reliant data types, from texts to images and beyond, including data from non-Euclidean geometries. In this paper, we present such a mechanism, designed to classify sequences of Symmetric Positive Definite matrices while preserving their Riemannian geometry throughout the analysis. We apply our method to automatic sleep staging on timeseries of EEG-derived covariance matrices from a standard dataset, obtaining high levels of stage-wise performance.
Code (2)
Tasks
EEGEEG based sleep stagingSleep Stage DetectionSleep StagingSimilar Papers 제목 키워드 기반
Privacy-Preserving Vision Transformer Using Images Encrypted with Restricted Random Permutation Matrices
We propose a novel method for privacy-preserving fine-tuning vision transformers (ViTs) with encrypted images. Conventional methods using encrypted images degrade model performance compared with that of using plain image…
Privacy PreservingMonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention
Transformers have achieved state-of-the-art performance across various tasks, but suffer from a notable quadratic complexity in sequence length due to the attention mechanism. In this work, we propose MonarchAttention --…
16k4kVMonarch: Efficient Video Diffusion Transformers with Structured Attention
The quadratic complexity of the attention mechanism severely limits the context scalability of Video Diffusion Transformers (DiTs). We find that the highly sparse spatio-temporal attention patterns exhibited in Video DiT…
Generating Long Sequences with Sparse Transformers
Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n…
DiversityImage GenerationLanguage ModellingOpen-Domain Question Answering+1Exact Sequence Classification with Hardmax Transformers
We prove that hardmax attention transformers perfectly classify datasets of $N$ labeled sequences in $\mathbb{R}^d$, $d\geq 2$. Specifically, given $N$ sequences with an arbitrary but finite length in $\mathbb{R}^d$, we …
Classification