Machine-in-the-Loop Rewriting for Creative Image Captioning
Machine-in-the-loop writing aims to enable humans to collaborate with models to complete their writing tasks more effectively. Prior work has found that providing humans a machine-written draft or sentence-level continuations has limited success since the generated text tends to deviate from humans' intention. To allow the user to retain control over the content, we train a rewriting model that, when prompted, modifies specified spans of text within the user's original draft to introduce descriptive and figurative elements locally in the text. We evaluate the model on its ability to collaborate with humans on the task of creative image captioning. On a user study through Amazon Mechanical Turk, our model is rated to be more helpful than a baseline infilling language model. In addition, third-party evaluation shows that users write more descriptive and figurative captions when collaborating with our model compared to completing the task alone.
Code (1)
Tasks
DescriptiveImage CaptioningLanguage ModelingLanguage ModellingSentenceSimilar Papers 제목 키워드 기반
Creative Captioning: An AI Grand Challenge Based on the Dixit Board Game
We propose a new class of "grand challenge" AI problems that we call creative captioning---generating clever, interesting, or abstract captions for images, as well as understanding such captions. Creative captioning draw…
Common Sense ReasoningRaidar: geneRative AI Detection viA Rewriting
We find that large language models (LLMs) are more likely to modify human-written text than AI-generated text when tasked with rewriting. This tendency arises because LLMs often perceive AI-generated text as high-quality…
Bringing back simplicity and lightliness into neural image captioning
Neural Image Captioning (NIC) or neural caption generation has attracted a lot of attention over the last few years. Describing an image with a natural language has been an emerging challenge in both fields of computer v…
Caption GenerationImage CaptioningMachine TranslationTranslation"It Felt Like Having a Second Mind": Investigating Human-AI Co-creativity in Prewriting with Large Language Models
Prewriting is the process of discovering and developing ideas before a first draft, which requires divergent thinking and often implies unstructured strategies such as diagramming, outlining, free-writing, etc. Although …
CreativeGAN: Editing Generative Adversarial Networks for Creative Design Synthesis
Modern machine learning techniques, such as deep neural networks, are transforming many disciplines ranging from image recognition to language understanding, by uncovering patterns in big data and making accurate predict…
Design SynthesisNovelty Detection