Latent Alignment of Procedural Concepts in Multimodal Recipes
We propose a novel alignment mechanism to deal with procedural reasoning on a newly released multimodal QA dataset, named RecipeQA. Our model is solving the textual cloze task which is a reading comprehension on a recipe containing images and instructions. We exploit the power of attention networks, cross-modal representations, and a latent alignment space between instructions and candidate answers to solve the problem. We introduce constrained max-pooling which refines the max-pooling operation on the alignment matrix to impose disjoint constraints among the outputs of the model. Our evaluation result indicates a 19\% improvement over the baselines.
Code (1)
Tasks
Question AnsweringReading ComprehensionSimilar Papers 제목 키워드 기반
A Recipe for Creating Multimodal Aligned Datasets for Sequential Tasks
Many high-level procedural tasks can be decomposed into sequences of instructions that vary in their order and choice of tools. In the cooking domain, the web offers many partially-overlapping text and video recipes (i.e…
DescriptiveRecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes
Understanding and reasoning about cooking recipes is a fruitful research direction towards enabling machines to interpret procedural text. In this work, we introduce RecipeQA, a dataset for multimodal comprehension of co…
Reading ComprehensionWhat does it take to bake a cake? The RecipeRef corpus and anaphora resolution in procedural text
Procedural text contains rich anaphoric phenomena, yet has not received much attention in NLP. To fill this gap, we investigate the textual properties of two types of procedural text, recipes and chemical patents, and ge…
Transfer LearningWhat does it take to bake a cake? The RecipeRef corpus and anaphora resolution in procedural text
Procedural text contains rich anaphoric phenomena yet has not received much attention in NLP. To fill this gap, we investigate the textual properties of two types of procedural text, recipes and chemical patents, and gen…
Transfer LearningStoryboarding of Recipes: Grounded Contextual Generation
Information need of humans is essentially multimodal in nature, enabling maximum exploitation of situated context. We introduce a dataset for sequential procedural (how-to) text generation from images in cooking domain. …
DecoderText Generation