SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors
Recent research efforts focus on reducing the computational and memory overheads of Large Language Models (LLMs) to make them feasible on resource-constrained devices. Despite advancements in compression techniques, non-linear operators like Softmax and Layernorm remain bottlenecks due to their sensitivity to quantization. We propose SoftmAP, a software-hardware co-design methodology that implements an integer-only low-precision Softmax using In-Memory Compute (IMC) hardware. Our method achieves up to three orders of magnitude improvement in the energy-delay product compared to A100 and RTX3090 GPUs, making LLMs more deployable without compromising performance.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationSensitivityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SOFTMAP: Sim2Real Soft Robot Forward Modeling via Topological Mesh Alignment and Physics Prior
While soft robot manipulators offer compelling advantages over rigid counterparts, including inherent compliance, safe human-robot interaction, and the ability to conform to complex geometries, accurate forward modeling …
Point CloudsLinear Symmetric Quantization of Neural Networks for Low-precision Integer Hardware
With the proliferation of specialized neural network processors that operate on low-precision integers, the performance of Deep Neural Network inference becomes increasingly dependent on the result of quantization. Despi…
object-detectionObject DetectionQuantizationDOS: Distilling Observable Softmaps of Zipfian Prototypes for Self-Supervised Point Representation
Recent advances in self-supervised learning (SSL) have shown tremendous potential for learning 3D point cloud representations without human annotations. However, SSL for 3D point clouds still faces critical challenges du…
Self-Supervised LearningSemantic Segmentation3D Object DetectionPoint CloudsKG-SoftMAP: Soft Knowledge-Graph Priors for Bayesian Network Structure Learning from Sparse Discrete Data
Learning Bayesian network (BN) structure from sparse discrete data is hard: when each instance records only a few variables, most variable pairs lack the joint observations needed for reliable scoring, and data-only meth…
NITI: Training Integer Neural Networks Using Integer-only Arithmetic
While integer arithmetic has been widely adopted for improved performance in deep quantized neural network inference, training remains a task primarily executed using floating point arithmetic. This is because both high …