Proximal Policy Optimization Learning based Control of Congested Freeway Traffic
This study proposes a delay-compensated feedback controller based on proximal policy optimization (PPO) reinforcement learning to stabilize traffic flow in the congested regime by manipulating the time-gap of adaptive cruise control-equipped (ACC-equipped) vehicles.The traffic dynamics on a freeway segment are governed by an Aw-Rascle-Zhang (ARZ) model, consisting of $2\times 2$ nonlinear first-order partial differential equations (PDEs).Inspired by the backstepping delay compensator [18] but different from whose complex segmented control scheme, the PPO control is composed of three feedbacks, namely the current traffic flow velocity, the current traffic flow density and previous one step control input. The control gains for the three feedbacks are learned from the interaction between the PPO and the numerical simulator of the traffic system without knowing the system dynamics. Numerical simulation experiments are designed to compare the Lyapunov control, the backstepping control and the PPO control. The results show that for a delay-free system, the PPO control has faster convergence rate and less control effort than the Lyapunov control. For a traffic system with input delay, the performance of the PPO controller is comparable to that of the Backstepping controller, even for the situation that the delay value does not match. However, the PPO is robust to parameter perturbations, while the Backstepping controller cannot stabilize a system where one of the parameters is disturbed by Gaussian noise.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Decentralized Cooperative Lane Changing at Freeway Weaving Areas Using Multi-Agent Deep Reinforcement Learning
Frequent lane changes during congestion at freeway bottlenecks such as merge and weaving areas further reduce roadway capacity. The emergence of deep reinforcement learning (RL) and connected and automated vehicle techno…
Deep Reinforcement LearningReinforcement Learning (RL)Freeway Merging in Congested Traffic based on Multipolicy Decision Making with Passive Actor Critic
Freeway merging in congested traffic is a significant challenge toward fully automated driving. Merging vehicles need to decide not only how to merge into a spot, but also where to merge. We present a method for the free…
Decision Makingreinforcement-learningReinforcement LearningReinforcement Learning (RL)On the Impact of the Capacity Drop Phenomenon for Freeway Traffic Flow Control
Capacity drop is an empirically observed phenomenon in vehicular traffic flow on freeways whereby, after a critical density is reached, a state of congestion sets in, but the freeway does not become decongested again unt…
Distributed Urban Freeway Traffic Optimization Considering Congestion Propagation
Effective traffic optimization strategies can improve the performance of transportation networks significantly. Most exiting works develop traffic optimization strategies depending on the local traffic states of congeste…
Decision-making for Autonomous Vehicles on Highway: Deep Reinforcement Learning with Continuous Action Horizon
Decision-making strategy for autonomous vehicles de-scribes a sequence of driving maneuvers to achieve a certain navigational mission. This paper utilizes the deep reinforcement learning (DRL) method to address the conti…
Autonomous VehiclesDecision MakingDeep Reinforcement LearningReinforcement Learning (RL)