How Well Can Transformers Emulate In-context Newton's Method?
Transformer-based models have demonstrated remarkable in-context learning capabilities, prompting extensive research into its underlying mechanisms. Recent studies have suggested that Transformers can implement first-order optimization algorithms for in-context learning and even second order ones for the case of linear regression. In this work, we study whether Transformers can perform higher order optimization methods, beyond the case of linear regression. We establish that linear attention Transformers with ReLU layers can approximate second order optimization algorithms for the task of logistic regression and achieve $\epsilon$ error with only a logarithmic to the error more layers. As a by-product we demonstrate the ability of even linear attention-only Transformers in implementing a single step of Newton's iteration for matrix inversion with merely two layers. These results suggest the ability of the Transformer architecture to implement complex algorithms, beyond gradient descent.
Code (0)
등록된 구현이 없습니다.
Tasks
In-Context LearningregressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression
Transformers excel at in-context learning (ICL) -- learning from demonstrations without parameter updates -- but how they do so remains a mystery. Recent work suggests that Transformers may internally run Gradient Descen…
In-Context LearningNewton Raphson Emulation Network for Highly Efficient Computation of Numerous Implied Volatilities
In finance, implied volatility is an important indicator that reflects the market situation immediately. Many practitioners estimate volatility using iteration methods, such as the Newton--Raphson (NR) method. However, i…
Deep LearningLooped Transformers as Programmable Computers
We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punchcard, consisting of instructions and me…
In-Context LearningAI-Newton: A Concept-Driven Physical Law Discovery System without Prior Physical Knowledge
Current limitations in human scientific discovery necessitate a new research paradigm. While advances in artificial intelligence (AI) offer a highly promising solution, enabling AI to emulate human-like scientific discov…
scientific discoveryA Closer Look at In-Context Learning under Distribution Shifts
In-context learning, a capability that enables a model to learn from input examples on the fly without necessitating weight updates, is a defining characteristic of large language models. In this work, we follow the sett…
In-Context Learning