On Taking Advantage of Opportunistic Meta-knowledge to Reduce Configuration Spaces for Automated Machine Learning
The automated machine learning (AutoML) process can require searching through complex configuration spaces of not only machine learning (ML) components and their hyperparameters but also ways of composing them together, i.e. forming ML pipelines. Optimisation efficiency and the model accuracy attainable for a fixed time budget suffer if this pipeline configuration space is excessively large. A key research question is whether it is both possible and practical to preemptively avoid costly evaluations of poorly performing ML pipelines by leveraging their historical performance for various ML tasks, i.e. meta-knowledge. The previous experience comes in the form of classifier/regressor accuracy rankings derived from either (1) a substantial but non-exhaustive number of pipeline evaluations made during historical AutoML runs, i.e. 'opportunistic' meta-knowledge, or (2) comprehensive cross-validated evaluations of classifiers/regressors with default hyperparameters, i.e. 'systematic' meta-knowledge. Numerous experiments with the AutoWeka4MCPS package suggest that (1) opportunistic/systematic meta-knowledge can improve ML outcomes, typically in line with how relevant that meta-knowledge is, and (2) configuration-space culling is optimal when it is neither too conservative nor too radical. However, the utility and impact of meta-knowledge depend critically on numerous facets of its generation and exploitation, warranting extensive analysis; these are often overlooked/underappreciated within AutoML and meta-learning literature. In particular, we observe strong sensitivity to the challenge' of a dataset, i.e. whether specificity in choosing a predictor leads to significantly better performance. Ultimately, identifying difficult' datasets, thus defined, is crucial to both generating informative meta-knowledge bases and understanding optimal search-space reduction strategies.
Code (1)
Tasks
AutoMLMeta-LearningSpecificitySimilar Papers 제목 키워드 기반
Exploiting Unfair Advantages: Investigating Opportunistic Trading in the NFT Market
As cryptocurrency evolved, new financial instruments, such as lending and borrowing protocols, currency exchanges, fungible and non-fungible tokens (NFT), staking and mining protocols have emerged. A financial ecosystem …
Exploring Opportunistic Meta-knowledge to Reduce Search Spaces for Automated Machine Learning
Machine learning (ML) pipeline composition and optimisation have been studied to seek multi-stage ML models, i.e. preprocessor-inclusive, that are both valid and well-performing. These processes typically require the des…
BIG-bench Machine LearningvalidMeta reinforcement learning as task inference
Humans achieve efficient learning by relying on prior knowledge about the structure of naturally occurring tasks. There is considerable interest in designing reinforcement learning (RL) algorithms with similar properties…
continuous-controlContinuous ControlMeta-LearningMeta Reinforcement Learning+3Stability versus Meta-stability in a Skin Microbiome Model
The skin microbiome plays an important role in the maintenance of a healthy skin. It is an ecosystem, composed of several species, competing for resources and interacting with the skin cells. Imbalance in the cutaneous m…
Deadline-Constrained Opportunistic Spectrum Access With Spectrum Handoff
This paper considers designing an optimal policy for deadline-constrained access in cognitive radio networks, where a secondary user needs to complete a packet transmission over the vacant spectrum within a delivery dead…