Perspective-corrected Spatial Referring Expression Generation for Human-Robot Interaction
Intelligent robots designed to interact with humans in real scenarios need to be able to refer to entities actively by natural language. In spatial referring expression generation, the ambiguity is unavoidable due to the diversity of reference frames, which will lead to an understanding gap between humans and robots. To narrow this gap, in this paper, we propose a novel perspective-corrected spatial referring expression generation (PcSREG) approach for human-robot interaction by considering the selection of reference frames. The task of referring expression generation is simplified into the process of generating diverse spatial relation units. First, we pick out all landmarks in these spatial relation units according to the entropy of preference and allow its updating through a stack model. Then all possible referring expressions are generated according to different reference frame strategies. Finally, we evaluate every expression using a probabilistic referring expression resolution model and find the best expression that satisfies both of the appropriateness and effectiveness. We implement the proposed approach on a robot system and empirical experiments show that our approach can generate more effective spatial referring expressions for practical applications.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityReferring ExpressionReferring expression generationRelationSimilar Papers 제목 키워드 기반
Grounding Language in Multi-Perspective Referential Communication
We introduce a task and dataset for referring expression generation and comprehension in multi-agent embodied environments. In this task, two agents in a shared scene must take into account one another's visual perspecti…
Referring ExpressionReferring expression generationRecurrent Instance Segmentation using Sequences of Referring Expressions
The goal of this work is to segment the objects in an image that are referred to by a sequence of linguistic descriptions (referring expressions). We propose a deep neural network with recurrent layers that output a sequ…
Instance SegmentationReferring ExpressionReferring Expression SegmentationSemantic SegmentationVision-Language Models Are Not Pragmatically Competent in Referring Expression Generation
Referring Expression Generation (REG) is a core task for evaluating the pragmatic competence of vision-language systems, requiring not only accurate semantic grounding but also adherence to principles of cooperative comm…
Referring ExpressionReferring expression generationGraphs and Spatial Relations in the Generation of Referring Expressions
Referring Expression Object Segmentation with Caption-Aware Consistency
Referring expressions are natural language descriptions that identify a particular object within a scene and are widely used in our daily conversations. In this work, we focus on segmenting the object in an image specifi…
Caption GenerationObjectReferring ExpressionReferring Expression Segmentation+2