Critic Guided Segmentation of Rewarding Objects in First-Person Views
This work discusses a learning approach to mask rewarding objects in images using sparse reward signals from an imitation learning dataset. For that, we train an Hourglass network using only feedback from a critic model. The Hourglass network learns to produce a mask to decrease the critic's score of a high score image and increase the critic's score of a low score image by swapping the masked areas between these two images. We trained the model on an imitation learning dataset from the NeurIPS 2020 MineRL Competition Track, where our model learned to mask rewarding objects in a complex interactive 3D environment with a sparse reward signal. This approach was part of the 1st place winning solution in this competition. Video demonstration and code: https://rebrand.ly/critic-guided-segmentation
Code (1)
Tasks
Imitation LearningSimilar Papers 제목 키워드 기반
Critic-Guided Learning to Segment Rewarding Objects in First-Person Views
We train a U-Net model to generate masks over reward-related objects in images. Our approach allows to train the U-Net model without explicit label information, but only using feedback from a critic model which learned t…
SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Images
Effectively grounding complex language to pixels in remote sensing (RS) images is a critical challenge for applications like disaster response and environmental monitoring. Current models can parse simple, single-target …
Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
Self-Rewarding Language Models propose an architecture in which the Large Language Models(LLMs) both generates responses and evaluates its own outputs via LLM-as-a-Judge prompting, dynamically improving its generative ca…
Mathematical ReasoningCode GenerationObject-Guided Instance Segmentation With Auxiliary Feature Refinement for Biological Images
Instance segmentation is of great importance for many biological applications, such as study of neural cell interactions, plant phenotyping, and quantitatively measuring how cells react to drug treatment. In this paper, …
Instance SegmentationObjectPlant PhenotypingSegmentation+1DR$^2$Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models
Reasoning segmentation is an emerging vision-language task that requires reasoning over intricate text queries to precisely segment objects. However, existing methods typically suffer from overthinking, generating verbos…
Multimodal ReasoningObject Localization