On-device Self-supervised Learning of Visual Perception Tasks aboard Hardware-limited Nano-quadrotors
Sub-\SI{50}{\gram} nano-drones are gaining momentum in both academia and industry. Their most compelling applications rely on onboard deep learning models for perception despite severe hardware constraints (\ie sub-\SI{100}{\milli\watt} processor). When deployed in unknown environments not represented in the training data, these models often underperform due to domain shift. To cope with this fundamental problem, we propose, for the first time, on-device learning aboard nano-drones, where the first part of the in-field mission is dedicated to self-supervised fine-tuning of a pre-trained convolutional neural network (CNN). Leveraging a real-world vision-based regression task, we thoroughly explore performance-cost trade-offs of the fine-tuning phase along three axes: \textit{i}) dataset size (more data increases the regression performance but requires more memory and longer computation); \textit{ii}) methodologies (\eg fine-tuning all model parameters vs. only a subset); and \textit{iii}) self-supervision strategy. Our approach demonstrates an improvement in mean absolute error up to 30\% compared to the pre-trained baseline, requiring only \SI{22}{\second} fine-tuning on an ultra-low-power GWT GAP9 System-on-Chip. Addressing the domain shift problem via on-device learning aboard nano-drones not only marks a novel result for hardware-limited robots but lays the ground for more general advancements for the entire robotics community.
Code (0)
등록된 구현이 없습니다.
Tasks
regressionSelf-Supervised LearningSimilar Papers 제목 키워드 기반
On-Device Self-Supervised Learning of Low-Latency Monocular Depth from Only Events
Event cameras provide low-latency perception for only milliwatts of power. This makes them highly suitable for resource-restricted, agile robots such as small flying drones. Self-supervised learning based on contrast max…
BenchmarkingComputational EfficiencyDepth EstimationDomain Adaptation+1Self-Supervised Video Representation Learning with Motion-Contrastive Perception
Visual-only self-supervised learning has achieved significant improvement in video representation learning. Existing related methods encourage models to learn video representations by utilizing contrastive learning or de…
Contrastive LearningRepresentation LearningSelf-Supervised LearningRobot Perception enables Complex Navigation Behavior via Self-Supervised Learning
Learning visuomotor control policies in robotic systems is a fundamental problem when aiming for long-term behavioral autonomy. Recent supervised-learning-based vision and motion perception systems, however, are often se…
Reinforcement Learning (RL)Self-Supervised LearningVisual LocalizationVisual OdometryAttendNets: Tiny Deep Image Recognition Neural Networks for the Edge via Visual Attention Condensers
While significant advances in deep learning has resulted in state-of-the-art performance across a large number of complex visual perception tasks, the widespread deployment of deep neural networks for TinyML applications…
Convolutional Visual Prompt for Robust Visual Perception
Vision models are often vulnerable to out-of-distribution (OOD) samples without adapting. While visual prompts offer a lightweight method of input-space adaptation for large-scale vision models, they rely on a high-dimen…
Test-time Adaptation