Towards Resolving Unidentifiability in Inverse Reinforcement Learning
We consider a setting for Inverse Reinforcement Learning (IRL) where the learner is extended with the ability to actively select multiple environments, observing an agent's behavior on each environment. We first demonstrate that if the learner can experiment with any transition dynamics on some fixed set of states and actions, then there exists an algorithm that reconstructs the agent's reward function to the fullest extent theoretically possible, and that requires only a small (logarithmic) number of experiments. We contrast this result to what is known about IRL in single fixed environments, namely that the true reward function is fundamentally unidentifiable. We then extend this setting to the more realistic case where the learner may not select any transition dynamic, but rather is restricted to some fixed set of environments that it may try. We connect the problem of maximizing the information derived from experiments to submodular function maximization and demonstrate that a greedy algorithm is near optimal (up to logarithmic factors). Finally, we empirically validate our algorithm on an environment inspired by behavioral psychology.
Code (0)
등록된 구현이 없습니다.
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Beyond Convergence: Identifiability of Machine Learning and Deep Learning Models
Machine learning (ML) and deep learning models are extensively used for parameter optimization and regression problems. However, not all inverse problems in ML are ``identifiable,'' indicating that model parameters may n…
Deep Learningparameter estimationExamining the impact of forcing function inputs on structural identifiability
For mathematical and experimental ease, models with time varying parameters are often simplified to assume constant parameters. However, this simplification can potentially lead to identifiability issues (lack of uniquen…
Mathematical ProofsUnidentifiability of System Dynamics: Conditions and Controller Design
How to make a dynamic system unidentifiable is an important but still open issue. It not only requires that the parameters of the systems but also the equivalent systems cannot be identified by any identification approac…
Robust estimation of tree structured Gaussian Graphical Model
Consider jointly Gaussian random variables whose conditional independence structure is specified by a graphical model. If we observe realizations of the variables, we can compute the covariance matrix, and it is well kno…
modelInverting Non-Injective Functions with Twin Neural Network Regression
Non-injective functions are not globally invertible. However, they can often be restricted to locally injective subdomains where the inversion is well-defined. In many settings a preferred solution can be selected even w…