Disentangling Recognition and Decision Regrets in Image-Based Reinforcement Learning

2024-09-19 · Alihan Hüyük, Arndt Ryo Koblitz, Atefeh Mohajeri, Matthew Andrews

In image-based reinforcement learning (RL), policies usually operate in two steps: first extracting lower-dimensional features from raw images (the "recognition" step), and then taking actions based on the extracted features (the "decision" step). Extracting features that are spuriously correlated with performance or irrelevant for decision-making can lead to poor generalization performance, known as observational overfitting in image-based RL. In such cases, it can be hard to quantify how much of the error can be attributed to poor feature extraction vs. poor decision-making. To disentangle the two sources of error, we introduce the notions of recognition regret and decision regret. Using these notions, we characterize and disambiguate the two distinct causes behind observational overfitting: over-specific representations, which include features that are not needed for optimal decision-making (leading to high decision regret), vs. under-specific representations, which only include a limited set of features that were spuriously correlated with performance during training (leading to high recognition regret). Finally, we provide illustrative examples of observational overfitting due to both over-specific and under-specific representations in maze environments and the Atari game Pong.

📄 PDF Abstract BibTeX arXiv:2409.13108

Code (0)

등록된 구현이 없습니다.

Tasks

Decision Making Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Learning in Markov Decision Processes under Constraints

2020-02-27 · Rahul Singh, Abhishek Gupta, Ness B. Shroff

We consider reinforcement learning (RL) in Markov Decision Processes in which an agent repeatedly interacts with an environment that is modeled by a controlled Markov process. At each time step $t$, it earns a reward, an…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Competitive Multi-Agent Deep Reinforcement Learning with Counterfactual Thinking

2019-08-13 · Yue Wang, Yao Wan, Chenwei Zhang, Lixin Cui 외

Counterfactual thinking describes a psychological phenomenon that people re-infer the possible results with different solutions about things that have already happened. It helps people to gain more experience from mistak…

counterfactual Decision Making Deep Reinforcement Learning Multi-agent Reinforcement Learning+3

Asymptotic Instance-Optimal Algorithms for Interactive Decision Making

2022-06-06 · Kefan Dong, Tengyu Ma

Past research on interactive decision making problems (bandits, reinforcement learning, etc.) mostly focuses on the minimax regret that measures the algorithm's performance on the hardest instance. However, an ideal algo…

Decision Making Multi-Armed Bandits reinforcement-learning Reinforcement Learning+1

Quantum Multi-Armed Bandits and Stochastic Linear Bandits Enjoy Logarithmic Regrets

2022-05-30 · Zongqi Wan, Zhijie Zhang, Tongyang Li, Jialin Zhang 외

Multi-arm bandit (MAB) and stochastic linear bandit (SLB) are important models in reinforcement learning, and it is well-known that classical algorithms for bandits with time horizon $T$ suffer $\Omega(\sqrt{T})$ regret.…

Multi-Armed Bandits reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Pareto Regret Analyses in Multi-objective Multi-armed Bandit

2022-12-01 · Mengfan Xu, Diego Klabjan

We study Pareto optimality in multi-objective multi-armed bandit by providing a formulation of adversarial multi-objective multi-armed bandit and defining its Pareto regrets that can be applied to both stochastic and adv…

Adversarial Attack