On- Device Information Extraction from Screenshots in form of tags
We propose a method to make mobile screenshots easily searchable. In this paper, we present the workflow in which we: 1) preprocessed a collection of screenshots, 2) identified script presentin image, 3) extracted unstructured text from images, 4) identifiedlanguage of the extracted text, 5) extracted keywords from the text, 6) identified tags based on image features, 7) expanded tag set by identifying related keywords, 8) inserted image tags with relevant images after ranking and indexed them to make it searchable on device. We made the pipeline which supports multiple languages and executed it on-device, which addressed privacy concerns. We developed novel architectures for components in the pipeline, optimized performance and memory for on-device computation. We observed from experimentation that the solution developed can reduce overall user effort and improve end user experience while searching, whose results are published.
Code (0)
등록된 구현이 없습니다.
Tasks
FormTAGSimilar Papers 제목 키워드 기반
ScreenSeg: On-Device Screenshot Layout Analysis
We propose a novel end-to-end solution that performs a Hierarchical Layout Analysis of screenshots and document images on resource constrained devices like mobilephones. Our approach segments entities like Grid, Image, T…
Image RetrievalStyle TransferText Extraction and Retrieval from Smartphone Screenshots: Building a Repository for Life in Media
Daily engagement in life experiences is increasingly interwoven with mobile device use. Screen capture at the scale of seconds is being used in behavioral studies and to implement "just-in-time" health interventions. The…
Image RetrievalOptical Character RecognitionOptical Character Recognition (OCR)RetrievalUnifying Multimodal Retrieval via Document Screenshot Embedding
In the real world, documents are organized in different formats and varied modalities. Traditional retrieval pipelines require tailored document parsing techniques and content extraction modules to prepare input for inde…
Language ModellingNatural QuestionsOptical Character Recognition (OCR)Retrieval+1On-Device Tag Generation for Unstructured Text
With the overwhelming transition to smart phones, storing important information in the form of unstructured text has become habitual to users of mobile devices. From grocery lists to drafts of emails and important speech…
FormTAGWorld KnowledgeOn Identifying Hashtags in Disaster Twitter Data
Tweet hashtags have the potential to improve the search for information during disaster events. However, there is a large number of disaster-related tweets that do not have any user-provided hashtags. Moreover, only a sm…
Disaster ResponseMulti-Task Learning