paper-with-me

홈 › Papers

MLHarness: A Scalable Benchmarking System for MLCommons

2021-11-09 · Yen-Hsiang Chang, Jianhao Pu, Wen-mei Hwu, JinJun Xiong

With the society's growing adoption of machine learning (ML) and deep learning (DL) for various intelligent solutions, it becomes increasingly imperative to standardize a common set of measures for ML/DL models with large scale open datasets under common development practices and resources so that people can benchmark and compare models quality and performance on a common ground. MLCommons has emerged recently as a driving force from both industry and academia to orchestrate such an effort. Despite its wide adoption as standardized benchmarks, MLCommons Inference has only included a limited number of ML/DL models (in fact seven models in total). This significantly limits the generality of MLCommons Inference's benchmarking results because there are many more novel ML/DL models from the research community, solving a wide range of problems with different inputs and outputs modalities. To address such a limitation, we propose MLHarness, a scalable benchmarking harness system for MLCommons Inference with three distinctive features: (1) it codifies the standard benchmark process as defined by MLCommons Inference including the models, datasets, DL frameworks, and software and hardware systems; (2) it provides an easy and declarative approach for model developers to contribute their models and datasets to MLCommons Inference; and (3) it includes the support of a wide range of models with varying inputs/outputs modalities so that we can scalably benchmark these models across different datasets, frameworks, and hardware systems. This harness system is developed on top of the MLModelScope system, and will be open sourced to the community. Our experimental results demonstrate the superior flexibility and scalability of this harness system for MLCommons Inference benchmarking.

📄 PDF Abstract BibTeX arXiv:2111.05231

Code (0)

등록된 구현이 없습니다.

Tasks

Benchmarking

Similar Papers 제목 키워드 기반

An MLCommons Scientific Benchmarks Ontology

2025-11-06 · Ben Hawks, Gregor von Laszewski, Matthew D. Sinclair, Marco Colombo 외 arxiv

Scientific machine learning research spans diverse domains and data modalities, yet existing benchmark efforts remain siloed and lack standardization. This makes novel and transformative applications of machine learning …

Improvements & Evaluations on the MLCommons CloudMask Benchmark

2024-03-07 · Varshitha Chennamsetti, Laiba Mehnaz, Dan Zhao, Banani Ghosh 외

In this paper, we report the performance benchmarking results of deep learning models on MLCommons' Science cloud-masking benchmark using a high-performance computing cluster at New York University (NYU): NYU Greene. MLC…

Benchmarking

MLCommons Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces

2026-05-11 · Srinivas Sridharan, Theodor-Adrian Badea, Andy Balogh, Bradford M. Beckmann 외 arxiv

The fast pace of artificial intelligence~(AI) innovation demands an agile methodology for observation, reproduction and optimization of distributed machine learning~(ML) workload behavior in production AI systems and ena…

MLPerf Automotive

2025-10-31 · Radoyeh Shojaei, Predrag Djurdjevic, Mostafa El-Khamy, James Goel 외 arxiv

We present MLPerf Automotive, the first standardized public benchmark for evaluating Machine Learning systems that are deployed for AI acceleration in automotive systems. Developed through a collaborative partnership bet…

2D Semantic Segmentation2D Object Detection3D Object Detection

MLCommons Cloud Masking Benchmark with Early Stopping

2023-12-11 · Varshitha Chennamsetti, Gregor von Laszewski, Ruochen Gu, Laiba Mehnaz 외

In this paper, we report on work performed for the MLCommons Science Working Group on the cloud masking benchmark. MLCommons is a consortium that develops and maintains several scientific benchmarks that aim to benefit d…