Learning Visual Representation from Human Interactions
Learning effective representations of visual data that generalize to a variety of downstream tasks has been a long quest for computer vision. Most representation learning approaches rely solely on visual data such as images or videos. In this paper, we explore a novel approach, where we use human interaction and attention cues to investigate whether we can learn better representations compared to visual-only representations. For this study, we collect a dataset of human interactions capturing body part movements and gaze in their daily lives. Our experiments show that our self-supervised representation that encodes interaction and attention cues outperforms a visual-only state-of-the-art method MoCo (He et al.,2020), on a variety of target tasks: scene classification (semantic), action recognition (temporal), depth estimation (geometric), dynamics prediction (physics) and walkable surface estimation (affordance).
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionDepth EstimationRepresentation LearningScene ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Human Semantic Representations of Social Interactions from Moving Shapes
Humans are social creatures who readily recognize various social interactions from simple display of moving shapes. While previous research has often focused on visual features, we examine what semantic representations t…
Do Deep Neural Networks Model Nonlinear Compositionality in the Neural Representation of Human-Object Interactions?
Visual scene understanding often requires the processing of human-object interactions. Here we seek to explore if and how well Deep Neural Network (DNN) models capture features similar to the brain's representation of hu…
Human-Object Interaction DetectionObjectScene UnderstandingRelational visual representations underlie human social interaction recognition
Humans effortlessly recognize social interactions from visual input. Attempts to model this ability have typically relied on generative inverse planning models, which make predictions by inverting a generative model of a…
Graph Neural NetworkWhat Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions
Learning effective representations of visual data that generalize to a variety of downstream tasks has been a long quest for computer vision. Most representation learning approaches rely solely on visual data such as ima…
Action RecognitionDepth EstimationRepresentation LearningScene ClassificationSelf-supervised visual learning from interactions with objects
Self-supervised learning (SSL) has revolutionized visual representation learning, but has not achieved the robustness of human vision. A reason for this could be that SSL does not leverage all the data available to human…
ObjectRepresentation LearningSelf-Supervised Learning