ViSaRL: Visual Reinforcement Learning Guided by Human Saliency

2024-03-16 · Anthony Liang, Jesse Thomason, Erdem Biyik

Training robots to perform complex control tasks from high-dimensional pixel input using reinforcement learning (RL) is sample-inefficient, because image observations are comprised primarily of task-irrelevant information. By contrast, humans are able to visually attend to task-relevant objects and areas. Based on this insight, we introduce Visual Saliency-Guided Reinforcement Learning (ViSaRL). Using ViSaRL to learn visual representations significantly improves the success rate, sample efficiency, and generalization of an RL agent on diverse tasks including DeepMind Control benchmark, robot manipulation in simulation and on a real robot. We present approaches for incorporating saliency into both CNN and Transformer-based encoders. We show that visual representations learned using ViSaRL are robust to various sources of visual perturbations including perceptual noise and scene variations. ViSaRL nearly doubles success rate on the real-robot tasks compared to the baseline which does not use saliency.

📄 PDF Abstract BibTeX arXiv:2403.10940

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Robot Manipulation

Saliency-R1: Incentivizing Unified Saliency Reasoning Capability in MLLM with Confidence-Guided Reinforcement Learning

2025-11-01 · Long Li, Shuichen Ji, Ziyang Luo, Zhihui Li 외 arxiv

Although multimodal large language models (MLLMs) excel in high-level vision-language reasoning, they lack inherent awareness of visual saliency, making it difficult to identify key visual elements. To bridge this gap, w…

Co-Salient Object Detection Reinforcement Learning Instance Segmentation

Relevance-guided Audio Visual Fusion for Video Saliency Prediction

2024-11-18 · Li Yu, Xuanzhe Sun, Pan Gao, Moncef Gabbouj

Audio data, often synchronized with video frames, plays a crucial role in guiding the audience's visual attention. Incorporating audio information into video saliency prediction tasks can enhance the prediction of human …

Prediction Saliency Prediction Video Saliency Prediction

What's Old is New Again: Classical Dimensionality Reduction for Efficient Saliency-Guided Biometric Attack Detection

2026-06-11 · Samuel Webster, Walter Scheirer arxiv

Saliency-guided training is a paradigm in visual recognition that encourages models to focus on the most relevant image regions during learning. While its application in biometric presentation attack detection (PAD) has …

Dimensionality Reduction Face Detection

Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learning

2022-09-16 · David Bertoin, Adil Zouitine, Mehdi Zouitine, Emmanuel Rachelson

Deep reinforcement learning policies, despite their outstanding efficiency in simulated visual control tasks, have shown disappointing ability to generalize across disturbances in the input training images. Changes in im…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Free-Lunch Saliency via Attention in Atari Agents

2019-08-07 · Dmitry Nikulin, Anastasia Ianina, Vladimir Aliev, Sergey Nikolenko

We propose a new approach to visualize saliency maps for deep neural network models and apply it to deep reinforcement learning agents trained on Atari environments. Our method adds an attention module that we call FLS (…

Decision Making Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1