Enabling Machine Learning-Ready HPC Ensembles with Merlin
With the growing complexity of computational and experimental facilities, many scientific researchers are turning to machine learning (ML) techniques to analyze large scale ensemble data. With complexities such as multi-component workflows, heterogeneous machine architectures, parallel file systems, and batch scheduling, care must be taken to facilitate this analysis in a high performance computing (HPC) environment. In this paper, we present Merlin, a workflow framework to enable large ML-friendly ensembles of scientific HPC simulations. By augmenting traditional HPC with distributed compute technologies, Merlin aims to lower the barrier for scientific subject matter experts to incorporate ML into their analysis. In addition to its design, we describe some example applications that Merlin has enabled on leadership-class HPC resources, such as the ML-augmented optimization of nuclear fusion experiments and the calibration of infectious disease models to study the progression of and possible mitigation strategies for COVID-19.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningSchedulingSimilar Papers 제목 키워드 기반
MerLin: A Discovery Engine for Photonic and Hybrid Quantum Machine Learning
Identifying where quantum models may offer practical benefits in near term quantum machine learning (QML) requires moving beyond isolated algorithmic proposals toward systematic and empirical exploration across models, d…
Quantum Machine LearningMerlin HugeCTR: GPU-accelerated Recommender System Training and Inference
In this talk, we introduce Merlin HugeCTR. Merlin HugeCTR is an open source, GPU-accelerated integration framework for click-through rate estimation. It optimizes both training and inference, whilst enabling model traini…
CPUGPURecommendation SystemsRetrievalMeta-Consolidation for Continual Learning
The ability to continuously learn and adapt itself to new tasks, without losing grasp of already acquired knowledge is a hallmark of biological learning systems, which current deep learning systems fall short of. In this…
Continual LearningModel-contrastive explanations through symbolic reasoning
Explaining how two machine learning classification models differ in their behaviour is gaining significance in eXplainable AI, given the increasing diffusion of learning-based decision support systems. Human decision-mak…
Counterfactual ExplanationExplainable artificial intelligencemodelMerlin: Deterministic Byte-Exact Deduplication for Lossless Context Optimization in Large Language Model Inference
Data-intensive applications, ranging from large-scale retrieval systems to advanced data pipelines, are increasingly bottlenecked by the processing of highly redundant text corpora. We present Merlin, a local-first, agno…