Multimodal Hierarchical Reinforcement Learning Policy for Task-Oriented Visual Dialog

2018-05-08 · WS 2018 7 · Jiaping Zhang, Tiancheng Zhao, Zhou Yu

Creating an intelligent conversational system that understands vision and language is one of the ultimate goals in Artificial Intelligence (AI)~\cite{winograd1972understanding}. Extensive research has focused on vision-to-language generation, however, limited research has touched on combining these two modalities in a goal-driven dialog context. We propose a multimodal hierarchical reinforcement learning framework that dynamically integrates vision and language for task-oriented visual dialog. The framework jointly learns the multimodal dialog state representation and the hierarchical dialog policy to improve both dialog task success and efficiency. We also propose a new technique, state adaptation, to integrate context awareness in the dialog state representation. We evaluate the proposed framework and the state adaptation technique in an image guessing game and achieve promising results.

📄 PDF Abstract BibTeX arXiv:1805.03257

Code (0)

등록된 구현이 없습니다.

Tasks

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Text Generation Visual Dialog

GoIRL: Graph-Oriented Inverse Reinforcement Learning for Multimodal Trajectory Prediction

2025-06-26 · Muleilan Pei, Shaoshuai Shi, Lu Zhang, Peiliang Li 외

Trajectory prediction for surrounding agents is a challenging task in autonomous driving due to its inherent uncertainty and underlying multimodality. Unlike prevailing data-driven methods that primarily rely on supervis…

Autonomous Driving Motion Forecasting Prediction Trajectory Prediction

Subgoal Discovery for Hierarchical Dialogue Policy Learning

2018-04-20 · EMNLP 2018 10 · Da Tang, Xiujun Li, Jianfeng Gao, Chong Wang 외

Developing agents to engage in complex goal-oriented dialogues is challenging partly because the main learning signals are very sparse in long conversations. In this paper, we propose a divide-and-conquer approach that d…

Hierarchical Reinforcement Learning Reinforcement Learning

GoChat: Goal-oriented Chatbots with Hierarchical Reinforcement Learning

2020-05-24 · Jianfeng Liu, Feiyang Pan, Ling Luo

A chatbot that converses like a human should be goal-oriented (i.e., be purposeful in conversation), which is beyond language generation. However, existing dialogue systems often heavily rely on cumbersome hand-crafted r…

Chatbot Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning+3

WorldRFT: Latent World Model Planning with Reinforcement Fine-Tuning for Autonomous Driving

2025-12-22 · Pengxuan Yang, Ben Lu, Zhongpu Xia, Chao Han 외 arxiv

Latent World Models enhance scene representation through temporal self-supervised learning, presenting a perception annotation-free paradigm for end-to-end autonomous driving. However, the reconstruction-oriented represe…

Self-Supervised Learning Representation Learning Reinforcement Learning Autonomous Driving

Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning

2025-07-09 · Jinmin He, Kai Li, Yifan Zang, Haobo Fu 외 arxiv

Offline multi-task reinforcement learning aims to learn a unified policy capable of solving multiple tasks using only pre-collected task-mixed datasets, without requiring any online interaction with the environment. Howe…

Reinforcement Learning