Energy Consumption of Dataframe Libraries for End-to-End Deep Learning Pipelines:A Comparative Analysis
This paper presents a detailed comparative analysis of the performance of three major Python data manipulation libraries - Pandas, Polars, and Dask - specifically when embedded within complete deep learning (DL) training and inference pipelines. The research bridges a gap in existing literature by studying how these libraries interact with substantial GPU workloads during critical phases like data loading, preprocessing, and batch feeding. The authors measured key performance indicators including runtime, memory usage, disk usage, and energy consumption (both CPU and GPU) across various machine learning models and datasets.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
In-depth Analysis On Parallel Processing Patterns for High-Performance Dataframes
The Data Science domain has expanded monumentally in both research and industry communities during the past decade, predominantly owing to the Big Data revolution. Artificial Intelligence (AI) and Machine Learning (ML) a…
Understanding the Energy Consumption of HPC Scale Artificial Intelligence
This paper contributes towards better understanding the energy consumption trade-offs of HPC scale Artificial Intelligence (AI), and more specifically Deep Learning (DL) algorithms. For this task we developed benchmark-t…
Comparative Analysis of Time Series Forecasting Approaches for Household Electricity Consumption Prediction
As a result of increasing population and globalization, the demand for energy has greatly risen. Therefore, accurate energy consumption forecasting has become an essential prerequisite for government planning, reducing p…
energy managementGaussian ProcessesManagementregression+3DataFrame QA: A Universal LLM Framework on DataFrame Question Answering Without Data Exposure
This paper introduces DataFrame question answering (QA), a novel task that utilizes large language models (LLMs) to generate Pandas queries for information retrieval and data analysis on dataframes, emphasizing safe and …
Information RetrievalQuestion AnsweringRetrievalComparative Design Space Exploration of Dense and Semi-Dense SLAM
SLAM has matured significantly over the past few years, and is beginning to appear in serious commercial products. While new SLAM systems are being proposed at every conference, evaluation is often restricted to qualitat…
Benchmarking