홈 › Papers

Vision-Language Models as a Source of Rewards

2023-12-14 · Kate Baumli, Satinder Baveja, Feryal Behbahani, Harris Chan, Gheorghe Comanici, Sebastian Flennerhag, Maxime Gazeau, Kristian Holsheimer, Dan Horgan, Michael Laskin, Clare Lyle, Hussain Masoom, Kay McKinney, Volodymyr Mnih, Alexander Neitz, Dmitry Nikulin, Fabio Pardo, Jack Parker-Holder, John Quan, Tim Rocktäschel, Himanshu Sahni, Tom Schaul, Yannick Schroecker, Stephen Spencer, Richie Steigerwald, Luyu Wang, Lei Zhang

Building generalist agents that can accomplish many goals in rich open-ended environments is one of the research frontiers for reinforcement learning. A key limiting factor for building generalist agents with RL has been the need for a large number of reward functions for achieving different goals. We investigate the feasibility of using off-the-shelf vision-language models, or VLMs, as sources of rewards for reinforcement learning agents. We show how rewards for visual achievement of a variety of language goals can be derived from the CLIP family of models, and used to train RL agents that can achieve a variety of language goals. We showcase this approach in two distinct visual domains and present a scaling trend showing how larger VLMs lead to more accurate rewards for visual goal achievement, which in turn produces more capable RL agents.

📄 PDF Abstract BibTeX arXiv:2312.09187

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning

Methods 이 논문이 사용한 방법론

CLIP Contrastive Language-Image Pre-training (CLIP), consisting of a simplified version of ConVIRT trained from scratch, is an efficient method of image representation learning…

Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning

2025-07-22 · Junhao Shen, Haiteng Zhao, Yuzhe Gu, Songyang Gao 외 arxiv

Enhancing large vision-language models (LVLMs) with visual slow-thinking reasoning is crucial for solving complex multimodal tasks. However, since LVLMs are mainly trained with vision-language alignment, it is difficult …

Reinforcement Learning Multimodal Reasoning

Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning

2024-02-08 · Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin 외

In this paper, we propose R$^3$: Learning Reasoning through Reverse Curriculum Reinforcement Learning (RL), a novel method that employs only outcome supervision to achieve the benefits of process supervision for large la…

GSM8K reinforcement-learning Reinforcement Learning (RL)

olmOCR 2: Unit Test Rewards for Document OCR

2025-10-22 · Jake Poznanski, Luca Soldaini, Kyle Lo arxiv

We present olmOCR 2, the latest in our family of powerful OCR systems for converting digitized print documents, like PDFs, into clean, naturally ordered plain text. olmOCR 2 is powered by olmOCR-2-7B-1025, a specialized,…

Reinforcement Learning

Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax

2026-05-14 · Zeli Su, Ziyin Zhang, Zhou Liu, Xuexian Song 외 arxiv

Extending large language models (LLMs) to low-resource languages often incurs an "alignment tax": improvements in the target language come at the cost of catastrophic forgetting in general capabilities. We argue that thi…

Reinforcement Learning Machine Translation Headline Generation

SDR: Set-Distance Rewards for Radiology Report Generation

2026-05-30 · Halil Ibrahim Gulluk, Max Van Puyvelde, Wim Van Criekinge, Olivier Gevaert arxiv

Reinforcement learning with verifiable rewards has rapidly advanced reasoning in vision--language models. However, for chest X-ray report generation, the standard rewards (i.e. exact-match accuracy and step-level process…