SCP: Soft Conditional Prompt Learning for Aerial Video Action Recognition
We present a new learning approach, Soft Conditional Prompt Learning (SCP), which leverages the strengths of prompt learning for aerial video action recognition. Our approach is designed to predict the action of each agent by helping the models focus on the descriptions or instructions associated with actions in the input videos for aerial/robot visual perception. Our formulation supports various prompts, including learnable prompts, auxiliary visual information, and large vision models to improve the recognition performance. We present a soft conditional prompt method that learns to dynamically generate prompts from a pool of prompt experts under different video inputs. By sharing the same objective with the task, our proposed SCP can optimize prompts that guide the model's predictions while explicitly learning input-invariant (prompt experts pool) and input-specific (data-dependent) prompt knowledge. In practice, we observe a 3.17-10.2% accuracy improvement on the aerial video datasets (Okutama, NECDrone), which consist of scenes with single-agent and multi-agent actions. We further evaluate our approach on ground camera videos to verify the effectiveness and generalization and achieve a 1.0-3.6% improvement on dataset SSV2. We integrate our method into the ROS2 as well.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionOptical Flow EstimationPrompt LearningTemporal Action LocalizationSimilar Papers 제목 키워드 기반
Human Action Recognition in Drone Videos using a Few Aerial Training Examples
Drones are enabling new forms of human actions surveillance due to their low cost and fast mobility. However, using deep neural networks for automatic aerial action recognition is difficult due to the need for a large nu…
Action ClassificationAction RecognitionTemporal Action LocalizationTracking in Aerial Hyperspectral Videos using Deep Kernelized Correlation Filters
Hyperspectral imaging holds enormous potential to improve the state-of-the-art in aerial vehicle tracking with low spatial and temporal resolutions. Recently, adaptive multi-modal hyperspectral sensors have attracted gro…
General ClassificationImage GenerationObject Trackingvehicle detectionDiffusionCinema: Text-to-Aerial Cinematography
We propose a novel Unmanned Aerial Vehicles (UAV) assisted creative capture system that leverages diffusion models to interpret high-level natural language prompts and automatically generate optimal flight trajectories f…
Soft Prompts for Evaluation: Measuring Conditional Distance of Capabilities
To help evaluate and understand the latent capabilities of language models, this paper introduces an approach using optimized input embeddings, or 'soft prompts,' as a metric of conditional distance between a model and a…
Red TeamingAero-World: Action-Conditioned Aerial Video Generation from Inertial Controls
Foundation video models produce visually impressive results, but their use in embodied AI remains limited because they are primarily trained on natural language rather than low-level control signals. This limitation is e…
Video Generation