NRGPT: An Energy-based Alternative for GPT
Generative Pre-trained Transformer (GPT) architectures are the most popular design for language modeling. Energy-based modeling is a different paradigm that views inference as a dynamical process operating on an energy landscape. We propose a minimal modification of the GPT setting to unify it with the EBM framework. The inference step of our model, which we call eNeRgy-GPT (NRGPT), is conceptualized as an exploration of the tokens on the energy landscape. We prove, and verify empirically, that under certain circumstances this exploration becomes gradient descent, although they don't necessarily lead to the best performing models. We demonstrate that our model performs well for simple language (Shakespeare dataset), algebraic ListOPS tasks, and richer settings such as OpenWebText language modeling. We also observe that our models may be more resistant to overfitting, doing so only during very long training.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Optimizing Bio-energy Supply Chain to Achieve Alternative Energy Targets
In response to global warming and the dwindling reservoirs of fossil fuels, Thailand has increasingly embraced alternative energy sources. Central to its energy development strategy is the Alternative Energy Development …
EnergyVis: Interactively Tracking and Exploring Energy Consumption for ML Models
The advent of larger machine learning (ML) models have improved state-of-the-art (SOTA) performance in various modeling tasks, ranging from computer vision to natural language. As ML models continue increasing in size, s…
e-Fold Cross-Validation for Recommender-System Evaluation
To combat the rising energy consumption of recommender systems we implement a novel alternative for k-fold cross validation. This alternative, named e-fold cross validation, aims to minimize the number of folds to achiev…
Recommendation SystemsValuing Distributed Energy Resources for Non-Wires Alternatives
Distributed energy resources (DER) as non-wires alternatives, regardless of owner, have the potential to reduce system operating costs and delay system upgrades. However, it is difficult to determine the appropriate econ…
Bilevel OptimizationRNA Alternative Splicing Prediction with Discrete Compositional Energy Network
A single gene can encode for different protein versions through a process called alternative splicing. Since proteins play major roles in cellular functions, aberrant splicing profiles can result in a variety of diseases…