Agents that Listen: High-Throughput Reinforcement Learning with Multiple Sensory Systems

2021-07-05 · Shashank Hegde, Anssi Kanervisto, Aleksei Petrenko

Humans and other intelligent animals evolved highly sophisticated perception systems that combine multiple sensory modalities. On the other hand, state-of-the-art artificial agents rely mostly on visual inputs or structured low-dimensional observations provided by instrumented environments. Learning to act based on combined visual and auditory inputs is still a new topic of research that has not been explored beyond simple scenarios. To facilitate progress in this area we introduce a new version of VizDoom simulator to create a highly efficient learning environment that provides raw audio observations. We study the performance of different model architectures in a series of tasks that require the agent to recognize sounds and execute instructions given in natural language. Finally, we train our agent to play the full game of Doom and find that it can consistently defeat a traditional vision-based adversary. We are currently in the process of merging the augmented simulator with the main ViZDoom code repository. Video demonstrations and experiment code can be found at https://sites.google.com/view/sound-rl.

📄 PDF Abstract BibTeX arXiv:2107.02195

Code (1)

hegde95/Agents_that_Listen

Tasks

Game of Doom reinforcement-learning Reinforcement Learning (RL)Vocal Bursts Intensity Prediction

Modeling Human-Like Color Naming Behavior in Context

2026-04-28 · Yuqing Zhang, Ecesu Ürker, Tessa Verhoef, Gemma Boleda 외 arxiv

Modeling the emergence of human-like lexicons in computational systems has advanced through the use of interacting neural agents, which simulate both learning and communicative pressures. The NeLLCom-Lex framework (Zhang…

Reinforcement Learning

Federated Reinforcement Learning for Uplink Centric Broadband Communication Optimization over Unlicensed Spectrum

2024-02-18 · Hui Zhou, Yansha Deng

To provide Uplink Centric Broadband Communication (UCBC), New Radio Unlicensed (NR-U) network has been standardized to exploit the unlicensed spectrum using Listen Before Talk (LBT) scheme to fairly coexist with the incu…

Adversarial Reinforcement Learning in Dynamic Channel Access and Power Control

2021-05-12 · Feng Wang, M. Cenk Gursoy, Senem Velipasalar

Deep reinforcement learning (DRL) has recently been used to perform efficient resource allocation in wireless communications. In this paper, the vulnerabilities of such DRL agents to adversarial attacks is studied. In pa…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worlds

2017-09-12 · Zhiyu Lin, Brent Harrison, Aaron Keech, Mark O. Riedl

We describe a method to use discrete human feedback to enhance the performance of deep learning agents in virtual three-dimensional environments by extending deep-reinforcement learning to model the confidence and consis…

Deep Reinforcement Learning Minecraft reinforcement-learning Reinforcement Learning+1

Biases for Emergent Communication in Multi-agent Reinforcement Learning

2019-12-11 · NeurIPS 2019 12 · Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou 외

We study the problem of emergent communication, in which language arises because speakers and listeners must communicate information in order to solve tasks. In temporally extended reinforcement learning domains, it has …

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)