R2GAN: Cross-Modal Recipe Retrieval With Generative Adversarial Network
Representing procedure text such as recipe for crossmodal retrieval is inherently a difficult problem, not mentioning to generate image from recipe for visualization. This paper studies a new version of GAN, named Recipe Retrieval Generative Adversarial Network (R2GAN), to explore the feasibility of generating image from procedure text for retrieval problem. The motivation of using GAN is twofold: learning compatible cross-modal features in an adversarial way, and explanation of search results by showing the images generated from recipes. The novelty of R2GAN comes from architecture design, specifically a GAN with one generator and dual discriminators is used, which makes the generation of image from recipe a feasible idea. Furthermore, empowered by the generated images, a two-level ranking loss in both embedding and image spaces are considered. These add-ons not only result in excellent retrieval performance, but also generate close-to-realistic food images useful for explaining ranking of recipes. On recipe1M dataset, R2GAN demonstrates high scalability to data size, outperforms all the existing approaches, and generates images intuitive for human to interpret the search results.
Code (0)
등록된 구현이 없습니다.
Tasks
Generative Adversarial NetworkRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Transformer-based Cross-Modal Recipe Embeddings with Large Batch Training
In this paper, we present a cross-modal recipe retrieval framework, Transformer-based Network for Large Batch Training (TNLBT), which is inspired by ACME~(Adversarial Cross-Modal Embedding) and H-T~(Hierarchical Transfor…
Contrastive LearningImage GenerationRetrievalSelf-Supervised LearningLearning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Images
Food computing is playing an increasingly important role in human daily life, and has found tremendous applications in guiding human behavior towards smart food consumption and healthy lifestyle. An important task under …
Cross-Modal RetrievalNutritionRetrievalTranslation+1Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning
Cross-modal recipe retrieval has recently gained substantial attention due to the importance of food in people's lives, as well as the availability of vast amounts of digital cooking recipes and food images to train mach…
Cross-Modal RetrievalRetrievalSelf-Supervised LearningRecipe2Vec: Multi-modal Recipe Representation Learning with Graph Neural Networks
Learning effective recipe representations is essential in food studies. Unlike what has been developed for image-based recipe retrieval or learning structural text embeddings, the combined effect of multi-modal informati…
Adversarial AttackGraph Neural NetworkNode ClassificationRepresentation Learning+1Cross-Modal Retrieval and Synthesis (X-MRS): Closing the Modality Gap in Shared Representation Learning
Computational food analysis (CFA) naturally requires multi-modal evidence of a particular food, e.g., images, recipe text, etc. A key to making CFA possible is multi-modal shared representation learning, which aims to cr…
Cross-Modal RetrievalImage GenerationRepresentation LearningRetrieval