Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation
Distributed dataflow systems like Spark and Flink enable the use of clusters for scalable data analytics. While runtime prediction models can be used to initially select appropriate cluster resources given target runtimes, the actual runtime performance of dataflow jobs depends on several factors and varies over time. Yet, in many situations, dynamic scaling can be used to meet formulated runtime targets despite significant performance variance. This paper presents Enel, a novel dynamic scaling approach that uses message propagation on an attributed graph to model dataflow jobs and, thus, allows for deriving effective rescaling decisions. For this, Enel incorporates descriptive properties that capture the respective execution context, considers statistics from individual dataflow tasks, and propagates predictions through the job graph to eventually find an optimized new scale-out. Our evaluation of Enel with four iterative Spark jobs shows that our approach is able to identify effective rescaling actions, reacting for instance to node failures, and can be reused across different execution contexts.
Code (1)
Tasks
DescriptiveSimilar Papers 제목 키워드 기반
OpenELM: An Efficient Language Model Family with Open Training and Inference Framework
The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and enabling investigations into data and model biases, as well as potential…
Language ModelingLanguage ModellingTrajSceneLLM: A Multimodal Perspective on Semantic GPS Trajectory Analysis
GPS trajectory data reveals valuable patterns of human mobility and urban dynamics, supporting a variety of spatial applications. However, traditional methods often struggle to extract deep semantic representations and i…
Temporal SequencesFrom the Culinary to the Political Meaning of ``quenelle'' : Using Topic Models For Identifying Novel Senses (De la quenelle culinaire \`a la quenelle politique : identification de changements s\'emantiques \`a l'aide des Topic Models) [in French]
SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generation
Dynamic scenes contain intricate spatio-temporal information, crucial for mobile robots, UAVs, and autonomous driving systems to make informed decisions. Parsing these scenes into semantic triplets <Subject-Predicate-Obj…
Autonomous DrivingGraph GenerationScene Graph GenerationData driven localized wave solution of the Fokas-Lenells equation using modified PINN
We investigate data driven localized wave solutions of the Fokas-Lenells equation by using physics informed neural network(PINN). We improve basic PINN by incorporating control parameters into the residual loss function.…