Why bother with geometry? On the relevance of linear decompositions of Transformer embeddings

2023-10-10 · Timothee Mickus, Raúl Vázquez

A recent body of work has demonstrated that Transformer embeddings can be linearly decomposed into well-defined sums of factors, that can in turn be related to specific network inputs or components. There is however still a dearth of work studying whether these mathematical reformulations are empirically meaningful. In the present work, we study representations from machine-translation decoders using two of such embedding decomposition methods. Our results indicate that, while decomposition-derived indicators effectively correlate with model performance, variation across different runs suggests a more nuanced take on this question. The high variability of our measurements indicate that geometry reflects model-specific characteristics more than it does sentence-specific computations, and that similar training conditions do not guarantee similar vector spaces.

📄 PDF Abstract BibTeX arXiv:2310.06977

Code (1)

timotheemickus/seq2seq-splat 공식 구현 pytorch

Tasks

Machine TranslationSentence

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Position-Wise Feed-Forward Layer 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression

2026-09-22 · Baher Mohammad, Ammar Ali, Stamatios Lefkimmiatis hf

Transformer architectures exhibit cross-layer redundancies, yet post-training compression pipelines typically optimize layers in isolation or rely on heuristic grouping strategies that disregard layer-specific activation…

Geometry-Aware Video Quality Assessment for Dynamic Digital Human

2023-10-24 · ZiCheng Zhang, Yingjie Zhou, Wei Sun, Xiongkuo Min 외

Dynamic Digital Humans (DDHs) are 3D digital models that are animated using predefined motions and are inevitably bothered by noise/shift during the generation process and compression distortion during the transmission p…

Attribute Video Quality Assessment Visual Question Answering (VQA)

On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces

2026-07-08 · Chethan Krishnamurthy Ramanaik, Tobias Callies, Michael Hecht, Eirini Ntoutsi arxiv

Adversarial vulnerability in deep neural networks (DNNs) has been studied from the perspectives of decision-boundary geometry, feature robustness, input-output Jacobians, and the instability of inverse problems. Here, we…

Tensor Decompositions in Deep Learning

2020-02-26 · Davide Bacciu, Danilo P. Mandic

The paper surveys the topic of tensor decompositions in modern machine learning applications. It focuses on three active research topics of significant relevance for the community. After a brief review of consolidated wo…

BIG-bench Machine Learning Deep Learning

Tensor Decompositions: A New Concept in Brain Data Analysis?

2013-05-02 · Andrzej Cichocki

Matrix factorizations and their extensions to tensor factorizations and decompositions have become prominent techniques for linear and multilinear blind source separation (BSS), especially multiway Independent Component …

blind source separation Classification Clustering Dimensionality Reduction+2