Enhancing Cluster Scheduling in HPC: A Continuous Transfer Learning for Real-Time Optimization

2025-09-22 · Leszek Sliwko, Jolanta Mizera-Pietraszko arxiv

This study presents a machine learning-assisted approach to optimize task scheduling in cluster systems, focusing on node-affinity constraints. Traditional schedulers like Kubernetes struggle with real-time adaptability, whereas the proposed continuous transfer learning model evolves dynamically during operations, minimizing retraining needs. Evaluated on Google Cluster Data, the model achieves over 99% accuracy, reducing computational overhead and improving scheduling latency for constrained tasks. This scalable solution enables real-time optimization, advancing machine learning integration in cluster management and paving the way for future adaptive scheduling strategies.

📄 PDF Abstract BibTeX arXiv:2509.22701

Code (0)

등록된 구현이 없습니다.

Tasks

Transfer Learning

TDMA in Adaptive Resonant Beam Charging for IoT Devices

2018-09-24

Resonant beam charging (RBC) can realize wireless power transfer (WPT) from a transmitter to multiple receivers via resonant beams. The adaptive RBC (ARBC) can effectively improve its energy utilization. In order to supp…

Scheduling

Network Contention-Aware Cluster Scheduling with Reinforcement Learning

2023-10-31 · Junyeol Ryu, Jeongyoon Eo

With continuous advances in deep learning, distributed training is becoming common in GPU clusters. Specifically, for emerging workloads with diverse amounts, ratios, and patterns of communication, we observe that networ…

GPU reinforcement-learning Reinforcement Learning Scheduling

Learning Scheduling Algorithms for Data Processing Clusters

2018-10-03 · Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zili Meng 외

Efficiently scheduling data processing jobs on distributed compute clusters requires complex algorithms. Current systems, however, use simple generalized heuristics and ignore workload characteristics, since developing a…

Reinforcement Learning Reinforcement Learning (RL)Scheduling

Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning

2026-06-21 · Zicheng Xu, Ruixuan Zhang, Yu-Neng Chuang, Xiuyi Lou 외 arxiv

Large Language Models (LLMs) achieve remarkable reasoning capabilities through reinforcement learning (RL) post-training. However, existing RL post-training commonly relies on uniform data sampling, which ignores the sem…

Reinforcement Learning

Enhancing Kubernetes Automated Scheduling with Deep Learning and Reinforcement Techniques for Large-Scale Cloud Computing Optimization

2024-02-26 · Zheng Xu, Yulu Gong, Yanlin Zhou, Qiaozhi Bao 외

With the continuous expansion of the scale of cloud computing applications, artificial intelligence technologies such as Deep Learning and Reinforcement Learning have gradually become the key tools to solve the automated…

Cloud Computing Deep Learning reinforcement-learning Reinforcement Learning+1