홈 › Papers

Zero-Space Cost Fault Tolerance for Transformer-based Language Models on ReRAM

2024-01-22 · Bingbing Li, Geng Yuan, Zigeng Wang, Shaoyi Huang, Hongwu Peng, Payman Behnam, Wujie Wen, Hang Liu, Caiwen Ding

Resistive Random Access Memory (ReRAM) has emerged as a promising platform for deep neural networks (DNNs) due to its support for parallel in-situ matrix-vector multiplication. However, hardware failures, such as stuck-at-fault defects, can result in significant prediction errors during model inference. While additional crossbars can be used to address these failures, they come with storage overhead and are not efficient in terms of space, energy, and cost. In this paper, we propose a fault protection mechanism that incurs zero space cost. Our approach includes: 1) differentiable structure pruning of rows and columns to reduce model redundancy, 2) weight duplication and voting for robust output, and 3) embedding duplicated most significant bits (MSBs) into the model weight. We evaluate our method on nine tasks of the GLUE benchmark with the BERT model, and experimental results prove its effectiveness.

📄 PDF Abstract BibTeX arXiv:2401.11664

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Residual Connection 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Pruning 설명 없음

Zero Memory Overhead Approach for Protecting Vision Transformer Parameters

2025-07-04 · Fereshteh Baradaran, Mohsen Raji, Azadeh Baradaran, Arezoo Baradaran 외 arxiv

Vision Transformers (ViTs) have demonstrated superior performance over Convolutional Neural Networks (CNNs) in various vision-related tasks such as classification, object detection, and segmentation due to their use of s…

Autonomous Driving Object Detection

Computational Efficiency

Zero-Space Cost Fault Tolerance for Transformer-based Language Models on ReRAM

Code (0)

Methods 이 논문이 사용한 방법론

Similar Papers 제목 키워드 기반

Zero Memory Overhead Approach for Protecting Vision Transformer Parameters

FT-Transformer: Resilient and Reliable Transformer with End-to-End Fault Tolerant Attention

Byzantine Fault-Tolerance in Federated Local SGD under 2f-Redundancy

Custom Algorithm-based Fault Tolerance for Attention Layers in Transformers

Exploring Winograd Convolution for Cost-effective Neural Network Fault Tolerance