Bilinear value networks
The dominant framework for off-policy multi-goal reinforcement learning involves estimating goal conditioned Q-value function. When learning to achieve multiple goals, data efficiency is intimately connected with the generalization of the Q-function to new goals. The de-facto paradigm is to approximate Q(s, a, g) using monolithic neural networks. To improve the generalization of the Q-function, we propose a bilinear decomposition that represents the Q-value via a low-rank approximation in the form of a dot product between two vector fields. The first vector field, f(s, a), captures the environment's local dynamics at the state s; whereas the second component, {\phi}(s, g), captures the global relationship between the current state and the goal. We show that our bilinear decomposition scheme substantially improves data efficiency, and has superior transfer to out-of-distribution goals compared to prior methods. Empirical evidence is provided on the simulated Fetch robot task-suite and dexterous manipulation with a Shadow hand.
Code (1)
Tasks
Multi-Goal Reinforcement LearningSimilar Papers 제목 키워드 기반
Robust Value Function Approximation Using Bilinear Programming
Existing value function approximation methods have been successfully used in many applications, but they often lack useful a priori error bounds. We propose approximate bilinear programming, a new formulation of value fu…
Recurrent Bilinear Optimization for Binary Neural Networks
Binary Neural Networks (BNNs) show great promise for real-world embedded devices. As one of the critical steps to achieve a powerful BNN, the scale factor calculation plays an essential role in reducing the performance g…
object-detectionObject DetectionBackpropagation with N-D Vector-Valued Neurons Using Arbitrary Bilinear Products
Vector-valued neural learning has emerged as a promising direction in deep learning recently. Traditionally, training data for neural networks (NNs) are formulated as a vector of scalars; however, its performance may not…
DenoisingImage DenoisingA framework for fitting quadratic-bilinear systems with applications to models of electrical circuits
In this contribution, we propose a data-driven procedure to fit quadratic-bilinear surrogate models from data. Although the dynamics characterizing the original model are strongly nonlinear, we rely on lifting techniques…
Bilinear Convolution Decomposition for Causal RL Interpretability
Efforts to interpret reinforcement learning (RL) models often rely on high-level techniques such as attribution or probing, which provide only correlational insights and coarse causal control. This work proposes replacin…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)