Using Logical Specifications of Objectives in Multi-Objective Reinforcement Learning

2019-10-03 · Kolby Nottingham, Anand Balakrishnan, Jyotirmoy Deshmukh, David Wingate

It is notoriously difficult to control the behavior of reinforcement learning agents. Agents often learn to exploit the environment or reward signal and need to be retrained multiple times. The multi-objective reinforcement learning (MORL) framework separates a reward function into several objectives. An ideal MORL agent learns to generalize to novel combinations of objectives allowing for better control of an agent's behavior without requiring retraining. Many MORL approaches use a weight vector to parameterize the importance of each objective. However, this approach suffers from lack of expressiveness and interpretability. We propose using propositional logic to specify the importance of multiple objectives. By using a logic where predicates correspond directly to objectives, specifications are inherently more interpretable. Additionally the set of specifications that can be expressed with formal languages is a superset of what can be expressed by weight vectors. In this paper, we define a formal language based on propositional logic with quantitative semantics. We encode logical specifications using a recurrent neural network and show that MORL agents parameterized by these encodings are able to generalize to novel specifications over objectives and achieve performance comparable to single objective baselines.

📄 PDF Abstract BibTeX arXiv:1910.01723

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-Objective Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Closing the Intent-to-Behavior Gap via Fulfillment Priority Logic

2025-03-04 · Bassel El Mabsout, Abdelrahman Abdelgawad, Renato Mancuso

Practitioners designing reinforcement learning policies face a fundamental challenge: translating intended behavioral objectives into representative reward functions. This challenge stems from behavioral intent requiring…

continuous-control Continuous Control Multi-Objective Reinforcement Learning reinforcement-learning+1

Multi-Agent Reinforcement Learning Guided by Signal Temporal Logic Specifications

2023-06-11 · Jiangwei Wang, Shuo Yang, Ziyan An, Songyang Han 외

Reward design is a key component of deep reinforcement learning, yet some tasks and designer's objectives may be unnatural to define as a scalar cost function. Among the various techniques, formal methods integrated with…

Deep Reinforcement Learning Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

WFA-IRL: Inverse Reinforcement Learning of Autonomous Behaviors Encoded as Weighted Finite Automata

2021-03-10 · Tianyu Wang, Nikolay Atanasov

This paper presents a method for learning logical task specifications and cost functions from demonstrations. Constructing specifications by hand is challenging for complex objectives and constraints in autonomous system…

reinforcement-learning Reinforcement Learning (RL)

DistSPECTRL: Distributing Specifications in Multi-Agent Reinforcement Learning Systems

2022-06-28 · Joe Eappen, Suresh Jagannathan

While notable progress has been made in specifying and learning objectives for general cyber-physical systems, applying these methods to distributed multi-agent systems still pose significant challenges. Among these are …

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Instructing Goal-Conditioned Reinforcement Learning Agents with Temporal Logic Objectives

2023-09-21 · NeurIPS 2023 11

Goal-conditioned reinforcement learning (RL) is a powerful approach for learning general-purpose skills by reaching diverse goals. However, it has limitations when it comes to task-conditioned policies, where goals are s…