Python Workflows on HPC Systems
The recent successes and wide spread application of compute intensive machine learning and data analytics methods have been boosting the usage of the Python programming language on HPC systems. While Python provides many advantages for the users, it has not been designed with a focus on multi-user environments or parallel programming - making it quite challenging to maintain stable and secure Python workflows on a HPC system. In this paper, we analyze the key problems induced by the usage of Python on HPC clusters and sketch appropriate workarounds for efficiently maintaining multi-user Python software environments, securing and restricting resources of Python jobs and containing Python processes, while focusing on Deep Learning applications running on GPU clusters.
Code (0)
등록된 구현이 없습니다.
Tasks
GPUSimilar Papers 제목 키워드 기반
Integration of Scanning Probe Microscope with High-Performance Computing: fixed-policy and reward-driven workflows implementation
The rapid development of computation power and machine learning algorithms has paved the way for automating scientific discovery with a scanning probe microscope (SPM). The key elements towards operationalization of auto…
scientific discoveryCopy-on-Write Scoring: Application-Specific Agent Evaluations
Trustworthy deployment of LLM-based agents in software systems requires evaluating how they perform on application-specific workflows, with enough granularity to localize where they succeed and fail. Yet existing agent e…
Pynblint: a Static Analyzer for Python Jupyter Notebooks
Jupyter Notebook is the tool of choice of many data scientists in the early stages of ML workflows. The notebook format, however, has been criticized for inducing bad programming practices; indeed, researchers have alrea…
ORMIR_XCT: A Python package for high resolution peripheral quantitative computed tomography image processing
High resolution peripheral quantitative computed tomography (HR-pQCT) is an imaging technique capable of imaging trabecular bone in-vivo. HR-pQCT has a wide range of applications, primarily focused on bone to improve our…
Experiments in Agentic AI for Science
This paper details two novel frameworks for developing autonomous, agentic AI in scientific workflows. Both systems leverage a hybrid Local Body, Remote Brain architecture via Google Colab, utilizing Python-based local o…
Attribute ExtractionKnowledge Graphs