Adaptive Optimization for Enhanced Efficiency in Large-Scale Language Model Training

2024-12-06 · Jiajing Chen, Bingying Liu, Xiaoxuan Liao, Jia Gao, Hongye Zheng, Yue Li

With the rapid development of natural language processing technology, large-scale language models (LLM) have achieved remarkable results in a variety of tasks. However, how to effectively train these huge models and improve their performance and computational efficiency remains an important challenge. This paper proposes an improved method based on adaptive optimization algorithm, aiming to improve the training efficiency and final performance of LLM. Through comparative experiments on the SQuAD and GLUE data sets, the experimental results show that compared with traditional optimization algorithms (such as SGD, Momentum, AdaGrad, RMSProp and Adam), the adaptive optimization algorithm we proposed has better accuracy and F1 score. Both have achieved significant improvements, especially showed stronger training capabilities when processed large-scale texts and complex tasks. The research results verify the advantages of adaptive optimization algorithms in large-scale language model training and provide new ideas and directions for future optimization methods.

📄 PDF Abstract BibTeX arXiv:2412.04718

Code (0)

등록된 구현이 없습니다.

Tasks

Computational EfficiencyLanguage ModelingLanguage Modelling

Methods 이 논문이 사용한 방법론

RMSProp RMSProp is an unpublished adaptive learning rate optimizer proposed by Geoff Hinton. The motivation…

AdaGrad AdaGrad is a stochastic optimization method that adapts the learning rate to the parameters. It performs smaller updates for parameters associated with frequently occurring…

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Efficient Adaptive Federated Optimization

2024-10-10 · Su Hyeong Lee, Sidharth Sharma, Manzil Zaheer, Tian Li

Adaptive optimization is critical in federated learning, where enabling adaptivity on both the server and client sides has proven essential for achieving optimal performance. However, the scalability of such jointly adap…

Federated Learning

Learning from Loss Landscape: Generalizable Mixed-Precision Quantization via Adaptive Sharpness-Aware Gradient Aligning

2025-05-08 · Lianbo Ma, Jianlun Ma, Yuee Zhou, Guoyang Xie 외

Mixed Precision Quantization (MPQ) has become an essential technique for optimizing neural network by determining the optimal bitwidth per layer. Existing MPQ methods, however, face a major hurdle: they require a computa…

Quantization

Dynamic Cooperative MAC Optimization in RSU-Enhanced VANETs: A Distributed Approach

2024-02-14 · Zhou Zhang, Saman Atapattu, Yizhu Wang, Sumei Sun 외

This paper presents an optimization approach for cooperative Medium Access Control (MAC) techniques in Vehicular Ad Hoc Networks (VANETs) equipped with Roadside Unit (RSU) to enhance network throughput. Our method employ…

Collision Avoidance

Federated Graph Learning with Adaptive Importance-based Sampling

2024-09-23 · Anran Li, YuanYuan Chen, Chao Ren, Wenhan Wang 외

For privacy-preserving graph learning tasks involving distributed graph datasets, federated learning (FL)-based GCN (FedGCN) training is required. A key challenge for FedGCN is scaling to large-scale graphs, which typica…

Federated Learning Graph Learning Graph Sampling Privacy Preserving

LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis

2024-03-22 · Kevin Xie, Jonathan Lorraine, Tianshi Cao, Jun Gao 외

Recent text-to-3D generation approaches produce impressive 3D results but require time-consuming optimization that can take up to an hour per prompt. Amortized methods like ATT3D optimize multiple prompts simultaneously …

3D Generation Text to 3D