Target-dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robots

2021-07-02 · Shintaro Ishikawa, Komei Sugiura

Currently, domestic service robots have an insufficient ability to interact naturally through language. This is because understanding human instructions is complicated by various ambiguities and missing information. In existing methods, the referring expressions that specify the relationships between objects are insufficiently modeled. In this paper, we propose Target-dependent UNITER, which learns the relationship between the target object and other objects directly by focusing on the relevant regions within an image, rather than the whole image. Our method is an extension of the UNITER-based Transformer that can be pretrained on general-purpose datasets. We extend the UNITER approach by introducing a new architecture for handling the target candidates. Our model is validated on two standard datasets, and the results show that Target-dependent UNITER outperforms the baseline method in terms of classification accuracy.

📄 PDF Abstract BibTeX arXiv:2107.00811

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Golden Queue Managers 설명 없음

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

UNITER UNITER or UNiversal Image-TExt Representation model is a large-scale pre-trained model for joint multimodal embedding. It is pre-trained using four image-text datasets COCO,…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Adam 설명 없음

UNITER: Learning UNiversal Image-TExt Representations

2019-09-25 · Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy 외

Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are jointly processed for visual and textual understanding. In this paper, we introduce UNITER, a UNiversal I…

Image-text matching Image-text Retrieval Language Modeling Language Modelling+11

X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

2020-09-23 · EMNLP 2020 11 · Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi 외

Mirroring the success of masked language models, vision-and-language counterparts like ViLBERT, LXMERT and UNITER have achieved state of the art performance on a variety of multimodal discriminative tasks like visual que…

Image Captioning Image Generation Question Answering Visual Grounding+2

UNITER: UNiversal Image-TExt Representation Learning

2019-09-25 · ECCV 2020 8 · Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy 외

Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are simultaneously processed for joint visual and textual understanding. In this paper, we introduce UNITER, …

Image-text matching Image-text Retrieval Language Modeling Language Modelling+14

Switching Head-Tail Funnel UNITER for Dual Referring Expression Comprehension with Fetch-and-Carry Tasks

2023-07-14 · Ryosuke Korekata, Motonari Kambara, Yu Yoshida, Shintaro Ishikawa 외

This paper describes a domestic service robot (DSR) that fetches everyday objects and carries them to specified destinations according to free-form natural language instructions. Given an instruction such as "Move the bo…

Object Referring Expression Referring Expression Comprehension

UNITER-Based Situated Coreference Resolution with Rich Multimodal Input

2021-12-07 · Yichen Huang, Yuchen Wang, Yik-Cheung Tam

We present our work on the multimodal coreference resolution task of the Situated and Interactive Multimodal Conversation 2.0 (SIMMC 2.0) dataset as a part of the tenth Dialog System Technology Challenge (DSTC10). We pro…

coreference-resolution Coreference Resolution Object Visual Dialog