Iterated Piecewise Affine (IPA) Approximation for Language Modeling
In this work, we demonstrate the application of a first-order Taylor expansion to approximate a generic function $F: R^{n \times m} \to R^{n \times m}$ and utilize it in language modeling. To enhance the basic Taylor expansion, we introduce iteration and piecewise modeling, leading us to name the algorithm the Iterative Piecewise Affine (IPA) approximation. The final algorithm exhibits interesting resemblances to the Transformers decoder architecture. By comparing parameter arrangements in IPA and Transformers, we observe a strikingly similar performance, with IPA outperforming Transformers by 1.5\% in the next token prediction task with cross-entropy loss for smaller sequence lengths.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Multiplication-Free Transformer Training via Piecewise Affine Operations
Multiplications are responsible for most of the computational cost involved in neural network training and inference. Recent research has thus looked for ways to reduce the cost associated with them. Inspired by Mogami (…
Error-free approximation of explicit linear MPC through lattice piecewise affine expression
In this paper, the disjunctive and conjunctive lattice piecewise affine (PWA) approximations of explicit linear model predictive control (MPC) are proposed. The training data are generated uniformly in the domain of inte…
Model Predictive ControlError Bounds for Compositions of Piecewise Affine Approximations
Nonlinear expressions are often approximated by piecewise affine (PWA) functions to simplify analysis or reduce computational costs. To reduce computational complexity, multivariate functions can be represented as compos…
Model Predictive Control of Wind Turbines with Piecewise-Affine Power Coefficient Approximation
In this paper, an offset-free bilinear model predictive control approach for wind turbines is presented. State-of-the-art controllers employ different control loops for pitch angle and generator torque which switch depen…
Model Predictive ControlLattice piecewise affine approximation of explicit model predictive control with application to satellite attitude control
Satellite attitude cotrol is a crucial part of aerospace technology, and model predictive control(MPC) is one of the most promising controllers in this area, which will be less effective if real-time online optimization …
Model Predictive Control