Multi-view Disentanglement for Reinforcement Learning with Multiple Cameras

2024-04-22 · Mhairi Dunion, Stefano V. Albrecht

The performance of image-based Reinforcement Learning (RL) agents can vary depending on the position of the camera used to capture the images. Training on multiple cameras simultaneously, including a first-person egocentric camera, can leverage information from different camera perspectives to improve the performance of RL. However, hardware constraints may limit the availability of multiple cameras in real-world deployment. Additionally, cameras may become damaged in the real-world preventing access to all cameras that were used during training. To overcome these hardware constraints, we propose Multi-View Disentanglement (MVD), which uses multiple cameras to learn a policy that is robust to a reduction in the number of cameras to generalise to any single camera from the training set. Our approach is a self-supervised auxiliary task for RL that learns a disentangled representation from multiple cameras, with a shared representation that is aligned across all cameras to allow generalisation to a single camera, and a private representation that is camera-specific. We show experimentally that an RL agent trained on a single third-person camera is unable to learn an optimal policy in many control tasks; but, our approach, benefiting from multiple cameras during training, is able to solve the task using only the same single third-person camera.

📄 PDF Abstract BibTeX arXiv:2404.14064

Code (1)

uoe-agents/mvd 공식 구현 pytorch

Tasks

Disentanglement reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulation

2025-05-07 · Abdulaziz Almuzairee, Rohan Patil, Dwait Bhatt, Henrik I. Christensen

Vision is well-known for its use in manipulation, especially using visual servoing. To make it robust, multiple cameras are needed to expand the field of view. That is computationally challenging. Merging multiple views …

Disentanglement Lightweight Deployment Q-Learning

LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World

2026-05-06 · Nan Yang, Julian Straub, Fan Zhang, Richard Newcombe 외 arxiv

Tracking 3D human motion from egocentric multi-camera headset is challenged by severe egomotion, partial visibility or occlusions and lack of training data. Existing methods designed for monocular video often require sta…

Cross-Camera Cow Identification via Disentangled Representation Learning

2026-02-07 · Runcheng Wang, Yaru Chen, Guiguo Zhang, Honghua Jiang 외 arxiv

Precise identification of individual cows is a fundamental prerequisite for comprehensive digital management in smart livestock farming. While existing animal identification methods excel in controlled, single-camera set…

Representation Learning

Learning State Representations in Complex Systems with Multimodal Data

2018-11-27 · Pavel Solovev, Vladimir Aliev, Pavel Ostyakov, Gleb Sterkin 외

Representation learning becomes especially important for complex systems with multimodal data sources such as cameras or sensors. Recent advances in reinforcement learning and optimal control make it possible to design c…

Anomaly Detection Disentanglement Model-based Reinforcement Learning reinforcement-learning+3

Active Vision for Early Recognition of Human Actions

2020-06-01 · CVPR 2020 6 · Boyu Wang, Lihan Huang, Minh Hoai

We propose a method for early recognition of human actions, one that can take advantages of multiple cameras while satisfying the constraints due to limited communication bandwidth and processing power. Our method consid…

reinforcement-learning Reinforcement Learning (RL)