Reinforcement Learning Under Moral Uncertainty
An ambitious goal for machine learning is to create agents that behave ethically: The capacity to abide by human moral norms would greatly expand the context in which autonomous agents could be practically and safely deployed, e.g. fully autonomous vehicles will encounter charged moral decisions that complicate their deployment. While ethical agents could be trained by rewarding correct behavior under a specific moral theory (e.g. utilitarianism), there remains widespread disagreement about the nature of morality. Acknowledging such disagreement, recent work in moral philosophy proposes that ethical behavior requires acting under moral uncertainty, i.e. to take into account when acting that one's credence is split across several plausible ethical theories. This paper translates such insights to the field of reinforcement learning, proposes two training methods that realize different points among competing desiderata, and trains agents in simple environments to act under moral uncertainty. The results illustrate (1) how such uncertainty can help curb extreme behavior from commitment to single theories and (2) several technical complications arising from attempting to ground moral philosophy in RL (e.g. how can a principled trade-off between two competing but incomparable reward functions be reached). The aim is to catalyze progress towards morally-competent agents and highlight the potential of RL to contribute towards the computational grounding of moral philosophy.
Code (1)
Tasks
Autonomous VehiclesBIG-bench Machine LearningPhilosophyreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Dropouts in Confidence: Moral Uncertainty in Human-LLM Alignment
Humans display significant uncertainty when confronted with moral dilemmas, yet the extent of such uncertainty in machines and AI agents remains underexplored. Recent studies have confirmed the overly confident tendencie…
Addressing Moral Uncertainty using Large Language Models for Ethical Decision-Making
We present an ethical decision-making framework that refines a pre-trained reinforcement learning (RL) model using a task-agnostic ethical layer. Following initial training, the RL model undergoes ethical fine-tuning, wh…
Decision MakingEthicsLarge Language ModelNavigate+1Towards Theory-based Moral AI: Moral AI with Aggregating Models Based on Normative Ethical Theory
Moral AI has been studied in the fields of philosophy and artificial intelligence. Although most existing studies are only theoretical, recent developments in AI have made it increasingly necessary to implement AI with m…
EthicsPhilosophyThe Moral Burden of Ambiguity Aversion
In their article, "Egalitarianism under Severe Uncertainty", Philosophy and Public Affairs, 46:3, 2018, Thomas Rowe and Alex Voorhoeve develop an original moral decision theory for cases under uncertainty, called "plural…
EthicsPhilosophyRule-Based Moral Principles for Explaining Uncertainty in Natural Language Generation
Large language models (LLMs) are increasingly used in high-stakes settings, where explaining uncertainty is both technical and ethical. Probabilistic methods are often opaque and misaligned with expectations of transpare…