Audio Interval Retrieval using Convolutional Neural Networks
Modern streaming services are increasingly labeling videos based on their visual or audio content. This typically augments the use of technologies such as AI and ML by allowing to use natural speech for searching by keywords and video descriptions. Prior research has successfully provided a number of solutions for speech to text, in the case of a human speech, but this article aims to investigate possible solutions to retrieve sound events based on a natural language query, and estimate how effective and accurate they are. In this study, we specifically focus on the YamNet, AlexNet, and ResNet-50 pre-trained models to automatically classify audio samples using their respective melspectrograms into a number of predefined classes. The predefined classes can represent sounds associated with actions within a video fragment. Two tests are conducted to evaluate the performance of the models on two separate problems: audio classification and intervals retrieval based on a natural language query. Results show that the benchmarked models are comparable in terms of performance, with YamNet slightly outperforming the other two models. YamNet was able to classify single fixed-size audio samples with 92.7% accuracy and 68.75% precision while its average accuracy on intervals retrieval was 71.62% and precision was 41.95%. The investigated method may be embedded into an automated event marking architecture for streaming services.
Code (0)
등록된 구현이 없습니다.
Tasks
Audio ClassificationRetrievalSpeech-to-TextSimilar Papers 제목 키워드 기반
Music Artist Classification with Convolutional Recurrent Neural Networks
Previous attempts at music artist classification use frame level audio features which summarize frequency content within short intervals of time. Comparatively, more recent music information retrieval tasks take advantag…
Artist classificationClassificationGeneral ClassificationInformation Retrieval+2Segment Length Matters: A Study of Segment Lengths on Audio Fingerprinting Performance
Audio fingerprinting provides an identifiable representation of acoustic signals, which can be later used for identification and retrieval systems. To obtain a discriminative representation, the input audio is usually se…
Attention as a Perspective for Learning Tempo-invariant Audio Queries
Current models for audio--sheet music retrieval via multimodal embedding space learning use convolutional neural networks with a fixed-size window for the input audio. Depending on the tempo of a query performance, this …
RetrievalAudio-based Near-Duplicate Video Retrieval with Audio Similarity Learning
In this work, we address the problem of audio-based near-duplicate video retrieval. We propose the Audio Similarity Learning (AuSiL) approach that effectively captures temporal patterns of audio similarity between video …
RetrievalTransfer LearningTripletVideo RetrievalDeep Learning for MIR Tutorial
Deep Learning has become state of the art in visual computing and continuously emerges into the Music Information Retrieval (MIR) and audio retrieval domain. In order to bring attention to this topic we propose an introd…
Deep LearningEvent DetectionInformation RetrievalMusic Information Retrieval+2