Memory-based Language Models: An Efficient, Explainable, and Eco-friendly Approach to Large Language Modeling
We present memory-based language modeling as an efficient, eco-friendly alternative to deep neural network-based language modeling. It offers log-linearly scalable next-token prediction performance and strong memorization capabilities. Implementing fast approximations of k-nearest neighbor classification, memory-based language modeling leaves a relatively small ecological footprint both in training and in inference mode, as it relies fully on CPUs and attains low token latencies. Its internal workings are simple and fully transparent. We compare our implementation of memory-based language modeling, OLIFANT, with GPT-2 and GPT-Neo on next-token prediction accuracy, estimated emissions and speeds, and offer some deeper analyses of the model.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models
Mixed-precision quantization is a promising approach for compressing large language models under tight memory budgets. However, existing mixed-precision methods typically suffer from one of two limitations: they either r…
FFAA: Multimodal Large Language Model based Explainable Open-World Face Forgery Analysis Assistant
The rapid advancement of deepfake technologies has sparked widespread public concern, particularly as face forgery poses a serious threat to public information security. However, the unknown and diverse forgery technique…
DescriptiveFace SwappingLanguage ModelingLanguage Modelling+3Agents: An Open-source Framework for Autonomous Language Agents
Recent advances on large language models (LLMs) enable researchers and developers to build autonomous language agents that can automatically solve various tasks and interact with environments, humans, and other agents us…
Unlocking the Potential of Large Language Models for Explainable Recommendations
Generating user-friendly explanations regarding why an item is recommended has become increasingly common, largely due to advances in language generation technology, which can enhance user trust and facilitate more infor…
Decision MakingExplainable RecommendationRecommendation SystemsText GenerationBackward-Friendly Optimization: Training Large Language Models with Approximate Gradients under Memory Constraints
Full fine-tuning of Large Language Models (LLMs) is notoriously memory-intensive, primarily because conventional optimizers such as SGD or Adam assume access to exact gradients derived from cached activations. Existing s…