ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation
Automatic evaluations for natural language generation (NLG) conventionally rely on token-level or embedding-level comparisons with text references. This differs from human language processing, for which visual imagination often improves comprehension. In this work, we propose ImaginE, an imagination-based automatic evaluation metric for natural language generation. With the help of StableDiffusion, a state-of-the-art text-to-image generator, we automatically generate an image as the embodied imagination for the text snippet and compute the imagination similarity using contextual embeddings. Experiments spanning several text generation tasks demonstrate that adding machine-generated images with our ImaginE displays great potential in introducing multi-modal information into NLG evaluation, and improves existing automatic metrics' correlations with human similarity judgments in both reference-based and reference-free evaluation scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
nlg evaluationText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation
Automatic evaluations for natural language generation conventionally rely on token-level or embedding-level comparisons with the text references. This is different from human evaluation manners, in which people also form…
nlg evaluationText GenerationLearning to Imagine: Visually-Augmented Natural Language Generation
People often imagine relevant scenes to aid in the writing process. In this work, we aim to utilize visual information for composition in the same manner as humans. We propose a method, LIVE, that makes pre-trained langu…
SentenceText GenerationVOPE: Revisiting Hallucination of Vision-Language Models in Voluntary Imagination Task
Most research on hallucinations in Large Vision-Language Models (LVLMs) focuses on factual description tasks that prohibit any output absent from the image. However, little attention has been paid to hallucinations in vo…
Attacking the Trusted Imagination: Oracle-Level Integrity Attacks on Imagine-then-Act World Models
Many recent vision-language-action (VLA) policies adopt an imagine-then-act design. A world-action model (WAM) first imagines a short future as a latent trajectory z~, on which the action is then conditioned. We identify…
Learning model-based planning from scratch
Conventional wisdom holds that model-based planning is a powerful approach to sequential decision-making. It is often very challenging in practice, however, because while a model can be used to evaluate a plan, it does n…
continuous-controlContinuous ControlDecision Makingmodel+1