A Multimodal Dialogue System for Conversational Image Editing

2020-02-16 · Tzu-Hsiang Lin, Trung Bui, Doo Soon Kim, Jean Oh

In this paper, we present a multimodal dialogue system for Conversational Image Editing. We formulate our multimodal dialogue system as a Partially Observed Markov Decision Process (POMDP) and trained it with Deep Q-Network (DQN) and a user simulator. Our evaluation shows that the DQN policy outperforms a rule-based baseline policy, achieving 90\% success rate under high error rates. We also conducted a real user study and analyzed real user behavior.

📄 PDF Abstract BibTeX arXiv:2002.06484

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

ChatUMM: Robust Context Tracking for Conversational Interleaved Generation

2026-02-06 · Wenxun Dai, Zhiyuan Zhao, Yule Zhong, Yiji Cheng 외 arxiv

Unified multimodal models (UMMs) have achieved remarkable progress yet remain constrained by a single-turn interaction paradigm, effectively functioning as solvers for independent requests rather than assistants in conti…

Text-to-Image Generation multimodal generation

Conversational Image Editing: Incremental Intent Identification in a New Dialogue Task

2018-07-01 · WS 2018 7 · Ramesh Manuvinakurike, Trung Bui, Walter Chang, Kallirroi Georgila

We present {``}conversational image editing{''}, a novel real-world application domain combining dialogue, visual information, and the use of computer vision. We discuss the importance of dialogue incrementality in this …

General Classification

DialogPaint: A Dialog-based Image Editing Model

2023-03-17 · Jingxuan Wei, Shiyu Wu, Xin Jiang, Yequan Wang

We introduce DialogPaint, a novel framework that bridges conversational interactions with image editing, enabling users to modify images through natural dialogue. By integrating a dialogue model with the Stable Diffusion…

model Style Transfer

MConv: An Environment for Multimodal Conversational Search across Multiple Domains

2021-07-11 · SIGIR 2021 7 · Lizi Liao, Le Hong Long, Zheng Zhang, Minlie Huang 외

Although conversational search has become a hot topic in both dialogue research and IR community, the real breakthrough has been limited by the scale and quality of datasets available. To address this fundamental obstacl…

Conversational Recommendation Conversational Search Dialogue State Tracking Response Generation

Multimodal Dialogue Response Generation

2021-10-16 · ACL 2022 5 · Qingfeng Sun, Yujing Wang, Can Xu, Kai Zheng 외

Responsing with image has been recognized as an important capability for an intelligent conversational agent. Yet existing works only focus on exploring the multimodal dialogue models which depend on retrieval-based meth…

Dialogue Generation Response Generation Retrieval