A Hybrid Model for Combining Neural Image Caption and k-Nearest Neighbor Approach for Image Captioning
A hybrid model is proposed that integrates two popular image captioning methods to generate a text-based summary describing the contents of the image. The two image captioning models are the Neural Image Caption (NIC) and the k-nearest neighbor approach. These are trained individually on the training set. We extract a set of five features, from the validation set, for evaluating the results of the two models that in turn is used to train a logistic regression classifier. The BLEU-4 scores of the two models are compared for generating the binary-value ground truth for the logistic regression classifier. For the test set, the input images are first passed separately through the two models to generate the individual captions. The five-dimensional feature set extracted from the two models is passed to the logistic regression classifier to take a decision regarding the final caption generated which is the best of two captions generated by the models. Our implementation of the k-nearest neighbor model achieves a BLEU-4 score of 15.95 and the NIC model achieves a BLEU-4 score of 16.01, on the benchmark Flickr8k dataset. The proposed hybrid model is able to achieve a BLEU-4 score of 18.20 proving the validity of our approach.
Code (1)
Tasks
Image CaptioningregressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Exploring Nearest Neighbor Approaches for Image Captioning
We explore a variety of nearest neighbor baseline approaches for image captioning. These approaches find a set of nearest neighbor images in the training set from which a caption may be borrowed for the query image. We s…
Image CaptioningNearest Neighbor Normalization Improves Multimodal Retrieval
Multimodal models leverage large-scale pre-training to achieve strong but still imperfect performance on tasks such as image captioning, visual question answering, and cross-modal retrieval. In this paper, we present a s…
Cross-Modal RetrievalImage CaptioningImage RetrievalImage-text Retrieval+4Hybrid Approximate Nearest Neighbor Indexing and Search (HANNIS) for Large Descriptor Databases
In this paper, we present a novel method for efficient and effective retrieval of similar deep descriptors. Our new hybrid method for indexing and searching for the approximate nearest neighbors in high-dimensional large…
RetrievalCollaborative filtering, K-nearest neighbor and cosine similarity in home decor recommender systems
An architectural framework, based on collaborative filtering using K-nearest neighbor and cosine similarity, was developed and implemented to fit the requirements for the company DecorRaid. The aim of the paper is to tes…
Collaborative FilteringRecommendation SystemsYes, we CANN: Constrained Approximate Nearest Neighbors for local feature-based visual localization
Large-scale visual localization systems continue to rely on 3D point clouds built from image collections using structure-from-motion. While the 3D points in these models are represented using local image features, direct…
Image RetrievalRetrievalVisual Localization