Value Alignment Verification
As humans interact with autonomous agents to perform increasingly complicated, potentially risky tasks, it is important that humans can verify these agents' trustworthiness and efficiently evaluate their performance and correctness. In this paper we formalize the problem of value alignment verification: how can a human efficiently test whether the goals and behavior of another agent are aligned with the human's values? We explore several different value alignment verification settings and provide foundational theory regarding value alignment verification. We study alignment verification problems with idealized human testers that know their own reward function as well as value alignment verification problems where the human tester has implicit values. Our theoretical results and our empirical results in both a discrete grid navigation domain and a continuous autonomous driving domain demonstrate that it is possible to synthesize highly efficient and accurate value alignment verification tests for certifying the alignment of autonomous agents.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous DrivingSimilar Papers 제목 키워드 기반
Value Alignment Verification
As humans interact with autonomous agents to perform increasingly complicated, potentially risky tasks, it is important to be able to efficiently evaluate an agent's performance and correctness. In this paper we formaliz…
Autonomous DrivingHybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Making
Target-conditioned sequence models provide a simple interface for controllable offline decision making, but the requested target return can be an unreliable control signal, especially when the target return lies in under…
Decision MakingScalable Synthesis of Formally Verified Neural Value Function for Hamilton-Jacobi Reachability Analysis
Hamilton-Jacobi (HJ) reachability analysis provides a formal method for guaranteeing safety in constrained control problems. It synthesizes a value function to represent a long-term safe set called feasible region. Early…
Expected Value Alignment for Generative Reward Modeling in Formal Mathematics Verification
Large Language Models (LLMs) are increasingly used with formal interactive theorem provers such as Lean 4. Scaling these systems with reinforcement learning or search methods requires process reward models (PRMs) that ca…
Reinforcement LearningThe Shadow Self: Intrinsic Value Misalignment in Large Language Model Agents
Large language model (LLM) agents with extended autonomy unlock new capabilities, but also introduce heightened challenges for LLM safety. In particular, an LLM agent may pursue objectives that deviate from human values …