Code as Reward: Empowering Reinforcement Learning with VLMs

2024-02-07 · David Venuto, Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang, Ankit Anand

Pre-trained Vision-Language Models (VLMs) are able to understand visual concepts, describe and decompose complex tasks into sub-tasks, and provide feedback on task completion. In this paper, we aim to leverage these capabilities to support the training of reinforcement learning (RL) agents. In principle, VLMs are well suited for this purpose, as they can naturally analyze image-based observations and provide feedback (reward) on learning progress. However, inference in VLMs is computationally expensive, so querying them frequently to compute rewards would significantly slowdown the training of an RL agent. To address this challenge, we propose a framework named Code as Reward (VLM-CaR). VLM-CaR produces dense reward functions from VLMs through code generation, thereby significantly reducing the computational burden of querying the VLM directly. We show that the dense rewards generated through our approach are very accurate across a diverse set of discrete and continuous environments, and can be more effective in training RL policies than the original sparse environment rewards.

📄 PDF Abstract BibTeX arXiv:2402.04764

Code (0)

등록된 구현이 없습니다.

Tasks

Code Generation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Test-Time Reasoning Through Visual Human Preferences with VLMs and Soft Rewards

2025-03-25 · Alexander Gambashidze, Konstantin Sobolev, Andrey Kuznetsov, Ivan Oseledets

Can Visual Language Models (VLMs) effectively capture human visual preferences? This work addresses this question by training VLMs to think about preferences at test time, employing reinforcement learning methods inspire…

World Knowledge

Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning

2025-03-23 · Yufei Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao 외

Large Vision-Language Models (LVLMs) typically follow a two-stage training paradigm-pretraining and supervised fine-tuning. Recently, preference optimization, derived from the language domain, has emerged as an effective…

Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning

2023-10-19 · Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez 외

Reinforcement learning (RL) requires either manually specifying a reward function, which is often infeasible, or learning a reward model from a large amount of human feedback, which is often very expensive. We study a mo…

MuJoCo Prompt Engineering reinforcement-learning Reinforcement Learning (RL)+2

Vision-Language Models as a Source of Rewards

2023-12-14 · Kate Baumli, Satinder Baveja, Feryal Behbahani, Harris Chan 외

Building generalist agents that can accomplish many goals in rich open-ended environments is one of the research frontiers for reinforcement learning. A key limiting factor for building generalist agents with RL has been…

reinforcement-learning Reinforcement Learning

From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme

2025-12-31 · Xueyan Li, Yingyi Xue, Mengjie Jiang, Qingzi Zhu 외 arxiv

Generating humorous memes is a challenging multimodal task that moves beyond direct image-to-caption supervision. It requires a nuanced reasoning over visual content, contextual cues, and subjective humor. To bridge this…

Reinforcement Learning multimodal generation