Safe Multi-agent Reinforcement Learning with Natural Language Constraints

2024-05-30 · Ziyan Wang, Meng Fang, Tristan Tomilin, Fei Fang, Yali Du

The role of natural language constraints in Safe Multi-agent Reinforcement Learning (MARL) is crucial, yet often overlooked. While Safe MARL has vast potential, especially in fields like robotics and autonomous vehicles, its full potential is limited by the need to define constraints in pre-designed mathematical terms, which requires extensive domain expertise and reinforcement learning knowledge, hindering its broader adoption. To address this limitation and make Safe MARL more accessible and adaptable, we propose a novel approach named Safe Multi-agent Reinforcement Learning with Natural Language constraints (SMALL). Our method leverages fine-tuned language models to interpret and process free-form textual constraints, converting them into semantic embeddings that capture the essence of prohibited states and behaviours. These embeddings are then integrated into the multi-agent policy learning process, enabling agents to learn policies that minimize constraint violations while optimizing rewards. To evaluate the effectiveness of SMALL, we introduce the LaMaSafe, a multi-task benchmark designed to assess the performance of multiple agents in adhering to natural language constraints. Empirical evaluations across various environments demonstrate that SMALL achieves comparable rewards and significantly fewer constraint violations, highlighting its effectiveness in understanding and enforcing natural language constraints.

📄 PDF Abstract BibTeX arXiv:2405.20018

Code (0)

등록된 구현이 없습니다.

Tasks

Autonomous Vehicles Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

Safe Reinforcement Learning with Natural Language Constraints

2020-10-11 · NeurIPS 2021 12 · Tsung-Yen Yang, Michael Hu, Yinlam Chow, Peter J. Ramadge 외

While safe reinforcement learning (RL) holds great promise for many practical applications like robotics or autonomous cars, current approaches require specifying constraints in mathematical form. Such specifications dem…

Autonomous Navigation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Models

2024-01-15 · Xingzhou Lou, Junge Zhang, Ziyan Wang, Kaiqi Huang 외

Safe reinforcement learning (RL) agents accomplish given tasks while adhering to specific constraints. Employing constraints expressed via easily-understandable human language offers considerable potential for real-world…

Form Reinforcement Learning (RL)Safe Reinforcement Learning

SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control

2025-08-27 · Quanfeng Lu, Zhantao Ma, Shuai Zhong, Jin Wang 외 arxiv

The rapid advancement of large vision language models (LVLMs) and agent systems has heightened interest in mobile GUI agents that can reliably translate natural language into interface operations. Existing single-agent a…

Multi-agent Reinforcement Learning Mathematical Reasoning

Multi-Agent Reinforcement Learning Guided by Signal Temporal Logic Specifications

2023-06-11 · Jiangwei Wang, Shuo Yang, Ziyan An, Songyang Han 외

Reward design is a key component of deep reinforcement learning, yet some tasks and designer's objectives may be unnatural to define as a scalar cost function. Among the various techniques, formal methods integrated with…

Deep Reinforcement Learning Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds

2026-06-13 · Ömer Veysel Çağatan, Xuandong Zhao arxiv

Reward hacking, where AI systems exploit misspecified objectives to achieve high reward without satisfying intended goals, remains a central challenge in AI safety. Yet most known instances have been discovered post hoc …

Reinforcement Learning