paper-with-me

홈 › Papers

See, Hear, Explore: Curiosity via Audio-Visual Association

2020-07-07 · NeurIPS 2020 12 · Victoria Dean, Shubham Tulsiani, Abhinav Gupta

Exploration is one of the core challenges in reinforcement learning. A common formulation of curiosity-driven exploration uses the difference between the real future and the future predicted by a learned model. However, predicting the future is an inherently difficult task which can be ill-posed in the face of stochasticity. In this paper, we introduce an alternative form of curiosity that rewards novel associations between different senses. Our approach exploits multiple modalities to provide a stronger signal for more efficient exploration. Our method is inspired by the fact that, for humans, both sight and sound play a critical role in exploration. We present results on several Atari environments and Habitat (a photorealistic navigation simulator), showing the benefits of using an audio-visual association model for intrinsically guiding learning agents in the absence of external rewards. For videos and code, see https://vdean.github.io/audio-curiosity.html.

📄 PDF Abstract BibTeX arXiv:2007.03669

Code (1)

vdean/audio-curiosity 공식 구현 tf

Tasks

Efficient Exploration

Similar Papers 제목 키워드 기반

Learning Visual Styles from Audio-Visual Associations

2022-05-10 · Tingle Li, Yichen Liu, Andrew Owens, Hang Zhao

From the patter of rain to the crunch of snow, the sounds we hear often convey the visual textures that appear within a scene. In this paper, we present a method for learning visual styles from unlabeled audio-visual dat…

Image Stylization

What killed the cat? Towards a logical formalization of curiosity (and suspense, and surprise) in narratives

2024-10-11 · Florence Dupin de Saint-Cyr, Anne-Gwenn Bosser, Benjamin Callac, Eric Maisel

We provide a unified framework in which the three emotions at the heart of narrative tension (curiosity, suspense and surprise) are formalized. This framework is built on nonmonotonic reasoning which allows us to compact…

Learning What To Hear: Boosting Sound-Source Association For Robust Audiovisual Instance Segmentation

2025-09-26 · Jinbae Seo, Hyeongjun Kwon, Kwonyoung Kim, Jiyoung Lee 외 arxiv

Audiovisual instance segmentation (AVIS) requires accurately localizing and tracking sounding objects throughout video sequences. Existing methods suffer from visual bias stemming from two fundamental issues: uniform add…

Instance Segmentation

AV-Unified: A Unified Framework for Audio-visual Scene Understanding

2026-03-06 · Guangyao Li, Xin Wang, Wenwu Zhu arxiv

When humans perceive the world, they naturally integrate multiple audio-visual tasks within dynamic, real-world scenes. However, current works such as event localization, parsing, segmentation and question answering are …

Scene UnderstandingQuestion Answering

Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Sound

2025-11-15 · Dengming Zhang, Weitao You, Jingxiong Li, Weishen Lin 외 arxiv

Emotion understanding is critical for making Large Language Models (LLMs) more general, reliable, and aligned with humans. Art conveys emotion through the joint design of visual and auditory elements, yet most prior work…