EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance
In this work, we aim to analyze and optimize the EnCLAP framework, a state-of-the-art model in automated audio captioning. We investigate the impact of modifying the acoustic encoder components, explore pretraining with different dataset scales, and study the effectiveness of a reranking scheme. Through extensive experimentation and quantitative analysis of generated captions, we develop EnCLAP++, an enhanced version that significantly surpasses the original.
Code (1)
Tasks
AudioCapsAudio captioningLanguage ModellingRerankingSimilar Papers 제목 키워드 기반
EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning
We propose EnCLAP, a novel framework for automated audio captioning. EnCLAP employs two acoustic representation models, EnCodec and CLAP, along with a pretrained language model, BART. We also introduce a new training obj…
AudioCapsAudio captioningLanguage ModelingLanguage ModellingExpanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioning
In this technical report, we describe our submission to DCASE2024 Challenge Task6 (Automated Audio Captioning) and Task8 (Language-based Audio Retrieval). We develop our approach building upon the EnCLAP audio captioning…
Audio captioningRerankingRetrievalCLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer
Automated Audio Captioning (AAC) aims to describe the semantic contexts of general sounds, including acoustic events and scenes, by leveraging effective acoustic features. To enhance performance, an AAC method, EnCLAP, e…
Audio captioningLanguage ModelingLanguage ModellingQuantizationFrom Motion Signals to Insights: A Unified Framework for Student Behavior Analysis and Feedback in Physical Education Classes
Analyzing student behavior in educational scenarios is crucial for enhancing teaching quality and student engagement. Existing AI-based models often rely on classroom video footage to identify and analyze student behavio…
Activity RecognitionHuman Activity RecognitionMeasurements with Noise: Bayesian Optimization for Co-optimizing Noise and Property Discovery in Automated Experiments
We have developed a Bayesian optimization (BO) workflow that integrates intra-step noise optimization into automated experimental cycles. Traditional BO approaches in automated experiments focus on optimizing experimenta…
Bayesian Optimization