Improving Spatiotemporal Self-Supervision by Deep Reinforcement Learning
Self-supervised learning of convolutional neural networks can harness large amounts of cheap unlabeled data to train powerful feature representations. As surrogate task, we jointly address ordering of visual data in the spatial and temporal domain. The permutations of training samples, which are at the core of self-supervision by ordering, have so far been sampled randomly from a fixed preselected set. Based on deep reinforcement learning we propose a sampling policy that adapts to the state of the network, which is being trained. Therefore, new permutations are sampled according to their expected utility for updating the convolutional feature representation. Experimental evaluation on unsupervised and transfer learning tasks demonstrates competitive performance on standard benchmarks for image and video classification and nearest neighbor retrieval.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement LearningGeneral Classificationreinforcement-learningReinforcement LearningReinforcement Learning (RL)RetrievalSelf-Supervised LearningTransfer LearningVideo ClassificationSimilar Papers 제목 키워드 기반
Skip-Clip: Self-Supervised Spatiotemporal Representation Learning by Future Clip Order Ranking
Deep neural networks require collecting and annotating large amounts of data to train successfully. In order to alleviate the annotation bottleneck, we propose a novel self-supervised representation learning approach for…
Action RecognitionFuture predictionRepresentation LearningSelf-Supervised Action RecognitionLoss is its own Reward: Self-Supervision for Reinforcement Learning
Reinforcement learning optimizes policies for expected cumulative reward. Need the supervision be so narrow? Reward is delayed and sparse for many tasks, making it a difficult and impoverished signal for end-to-end optim…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Representation LearningSpatiotemporal Forecasting as Planning: A Model-Based Reinforcement Learning Approach with Generative World Models
To address the dual challenges of inherent stochasticity and non-differentiable metrics in physical spatiotemporal forecasting, we propose Spatiotemporal Forecasting as Planning (SFP), a new paradigm grounded in Model-Ba…
Reinforcement LearningWeakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
We introduce the task of weakly supervised learning for detecting human and object interactions in videos. Our task poses unique challenges as a system does not know what types of human-object interactions are present in…
Human-Object Interaction DetectionObjectSentenceWeakly-supervised LearningBKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos
Quantifying motion in 3D is important for studying the behavior of humans and other animals, but manual pose annotations are expensive and time-consuming to obtain. Self-supervised keypoint discovery is a promising strat…
Decoder