Pretrained LLM Adapted with LoRA as a Decision Transformer for Offline RL in Quantitative Trading
Developing effective quantitative trading strategies using reinforcement learning (RL) is challenging due to the high risks associated with online interaction with live financial markets. Consequently, offline RL, which leverages historical market data without additional exploration, becomes essential. However, existing offline RL methods often struggle to capture the complex temporal dependencies inherent in financial time series and may overfit to historical patterns. To address these challenges, we introduce a Decision Transformer (DT) initialized with pre-trained GPT-2 weights and fine-tuned using Low-Rank Adaptation (LoRA). This architecture leverages the generalization capabilities of pre-trained language models and the efficiency of LoRA to learn effective trading policies from expert trajectories solely from historical data. Our model performs competitively with established offline RL algorithms, including Conservative Q-Learning (CQL), Implicit Q-Learning (IQL), and Behavior Cloning (BC), as well as a baseline Decision Transformer with randomly initialized GPT-2 weights and LoRA. Empirical results demonstrate that our approach effectively learns from expert trajectories and secures superior rewards in certain trading scenarios, highlighting the effectiveness of integrating pre-trained language models and parameter-efficient fine-tuning in offline RL for quantitative trading. Replication code for our experiments is publicly available at https://github.com/syyunn/finrl-dt
Code (1)
Tasks
Offline RLparameter-efficient fine-tuningQ-LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
In-Context Curiosity: Distilling Exploration for Decision-Pretrained Transformers on Bandit Tasks
As large language models (LLMs) continue to grow in capability, there is increasing interest in incorporating them into decision-making tasks. A common pipeline for this is Decision-Pretrained Transformers (DPTs). Howeve…
Multi-Armed BanditsOnline Decision Transformer
Recent work has shown that offline reinforcement learning (RL) can be formulated as a sequence modeling problem (Chen et al., 2021; Janner et al., 2021) and solved via approaches similar to large-scale language modeling.…
D4RLEfficient ExplorationLanguage ModelingLanguage Modelling+1Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining
Large transformer models pretrained on offline reinforcement learning datasets have demonstrated remarkable in-context reinforcement learning (ICRL) capabilities, where they can make good decisions when prompted with int…
In-Context Reinforcement Learningreinforcement-learningReinforcement LearningThompson SamplingReinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers
Decision Transformers have recently emerged as a new and compelling paradigm for offline Reinforcement Learning (RL), completing a trajectory in an autoregressive way. While improvements have been made to overcome initia…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)M$^3$PC: Test-time Model Predictive Control for Pretrained Masked Trajectory Model
Recent work in Offline Reinforcement Learning (RL) has shown that a unified Transformer trained under a masked auto-encoding objective can effectively capture the relationships between different modalities (e.g., states,…
D4RLmodelModel Predictive ControlReinforcement Learning (RL)