PIDformer: Transformer Meets Control Theory

2024-02-25 · Tam Nguyen, César A. Uribe, Tan M. Nguyen, Richard G. Baraniuk

In this work, we address two main shortcomings of transformer architectures: input corruption and rank collapse in their output representation. We unveil self-attention as an autonomous state-space model that inherently promotes smoothness in its solutions, leading to lower-rank outputs and diminished representation capacity. Moreover, the steady-state solution of the model is sensitive to input perturbations. We incorporate a Proportional-Integral-Derivative (PID) closed-loop feedback control system with a reference point into the model to improve robustness and representation capacity. This integration aims to preserve high-frequency details while bolstering model stability, rendering it more noise-resilient. The resulting controlled state-space model is theoretically proven robust and adept at addressing the rank collapse. Motivated by this control framework, we derive a novel class of transformers, PID-controlled Transformer (PIDformer), aimed at improving robustness and mitigating the rank-collapse issue inherent in softmax transformers. We empirically evaluate the model for advantages and robustness against baseline transformers across various practical tasks, including object classification, image segmentation, and language modeling.

📄 PDF Abstract BibTeX arXiv:2402.15989

Code (0)

등록된 구현이 없습니다.

Tasks

Image SegmentationLanguage ModelingLanguage Modelling Semantic Segmentation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

A Hyper-Transformer model for Controllable Pareto Front Learning with Split Feasibility Constraints

2024-02-04 · Tran Anh Tuan, Nguyen Viet Dung, Tran Ngoc Thang

Controllable Pareto front learning (CPFL) approximates the Pareto solution set and then locates a Pareto optimal solution with respect to a given reference vector. However, decision-maker objectives were limited to a con…

FedControl: When Control Theory Meets Federated Learning

2022-05-27 · Adnan Ben Mansour, Gaia Carenini, Alexandre Duplessis, David Naccache

To date, the most popular federated learning algorithms use coordinate-wise averaging of the model parameters. We depart from this approach by differentiating client contributions according to the performance of local le…

Federated Learning

Data-driven Meets Geometric Control: Zero Dynamics, Subspace Stabilization, and Malicious Attacks

2022-01-10 · Federico Celi, Fabio Pasqualetti

Studying structural properties of linear dynamical systems through invariant subspaces is one of the key contributions of the geometric approach to system theory. In general, a model of the dynamics is required in order …

Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Control

2015-06-08 · Prashanth L. A., Cheng Jie, Michael Fu, Steve Marcus 외

Cumulative prospect theory (CPT) is known to model human decisions well, with substantial empirical evidence supporting this claim. CPT works by distorting probabilities and is more general than the classic expected util…

Prediction reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Exploring Human-AI Conceptual Alignment through the Prism of Chess

2025-10-29 · Semyon Lomasov, Judah Goldfeder, Mehmet Hamza Erol, Matthew So 외 arxiv

Do AI systems truly understand human concepts or merely mimic surface patterns? We investigate this through chess, where human creativity meets precise strategic concepts. Analyzing a 270M-parameter transformer that achi…