Adaptive Hyperparameter Optimization for Continual Learning Scenarios
Hyperparameter selection in continual learning scenarios is a challenging and underexplored aspect, especially in practical non-stationary environments. Traditional approaches, such as grid searches with held-out validation data from all tasks, are unrealistic for building accurate lifelong learning systems. This paper aims to explore the role of hyperparameter selection in continual learning and the necessity of continually and automatically tuning them according to the complexity of the task at hand. Hence, we propose leveraging the nature of sequence task learning to improve Hyperparameter Optimization efficiency. By using the functional analysis of variance-based techniques, we identify the most crucial hyperparameters that have an impact on performance. We demonstrate empirically that this approach, agnostic to continual scenarios and strategies, allows us to speed up hyperparameters optimization continually across tasks and exhibit robustness even in the face of varying sequential task orders. We believe that our findings can contribute to the advancement of continual learning methodologies towards more efficient, robust and adaptable models for real-world applications.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningHyperparameter OptimizationLifelong learningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Adaptive Variational Continual Learning via Task-Heuristic Modelling
Variational continual learning (VCL) is a turn-key learning algorithm that has state-of-the-art performance among the best continual learning models. In our work, we explore an extension of the generalized variational co…
Continual LearningModel OptimizationAdaptive Explainable Continual Learning Framework for Regression Problems with Focus on Power Forecasts
Compared with traditional deep learning techniques, continual learning enables deep neural networks to learn continually and adaptively. Deep neural networks have to learn new tasks and overcome forgetting the knowledge …
Continual LearningregressionAddressing Client Drift in Federated Continual Learning with Adaptive Optimization
Federated learning has been extensively studied and is the prevalent method for privacy-preserving distributed learning in edge devices. Correspondingly, continual learning is an emerging field targeted towards learning …
Continual LearningFederated LearningPrivacy PreservingMultilayer Neuromodulated Architectures for Memory-Constrained Online Continual Learning
We focus on the problem of how to achieve online continual learning under memory-constrained conditions where the input data may not be known \emph{a priori}. These constraints are relevant in edge computing scenarios. W…
Bayesian Optimizationclass-incremental learningClass Incremental LearningContinual Learning+4Efficient Online Continual Foundation Model Fine-Tuning for Predictive Process Monitoring
Predictive Process Monitoring (PPM) models are increasingly deployed in dynamic environments where concept drift causes the underlying process distribution to shift over time. While recent work has moved toward online co…
Continual Learning