Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices

2020-05-07 · Chunjie Luo, Xiwen He, Jianfeng Zhan, Lei Wang, Wanling Gao, Jiahui Dai

Due to increasing amounts of data and compute resources, deep learning achieves many successes in various domains. The application of deep learning on the mobile and embedded devices is taken more and more attentions, benchmarking and ranking the AI abilities of mobile and embedded devices becomes an urgent problem to be solved. Considering the model diversity and framework diversity, we propose a benchmark suite, AIoTBench, which focuses on the evaluation of the inference abilities of mobile and embedded devices. AIoTBench covers three typical heavy-weight networks: ResNet50, InceptionV3, DenseNet121, as well as three light-weight networks: SqueezeNet, MobileNetV2, MnasNet. Each network is implemented by three frameworks which are designed for mobile and embedded devices: Tensorflow Lite, Caffe2, Pytorch Mobile. To compare and rank the AI capabilities of the devices, we propose two unified metrics as the AI scores: Valid Images Per Second (VIPS) and Valid FLOPs Per Second (VOPS). Currently, we have compared and ranked 5 mobile devices using our benchmark. This list will be extended and updated soon after.

📄 PDF Abstract BibTeX arXiv:2005.05085

Code (0)

등록된 구현이 없습니다.

Tasks

BenchmarkingDiversityvalid

Methods 이 논문이 사용한 방법론

Concatenated Skip Connection A Concatenated Skip Connection is a type of skip connection that seeks to reuse features by concatenating them to new layers, allowing more information to be retained from…

Dense Block A Dense Block is a module used in convolutional neural networks that connects *all layers* (with matching feature-map sizes) directly with each other. It was originally…

XRP Customer Service Number +1-833-534-1729 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Auxiliary Classifier Auxiliary Classifiers are type of architectural component that seek to improve the convergence of very deep networks. They are classifier heads we attach to layers before the…

Inception-v3 Module Inception-v3 Module is an image block used in the Inception-v3 architecture. This architecture is used on the coarsest (8 ×…

Inception-v3 Inception-v3 is a convolutional neural network architecture from the Inception family that makes several improvements including using [Label…

Bottleneck Residual Block A Bottleneck Residual Block is a variant of the residual block that utilises 1x1 convolutions to create a bottleneck. The…

Benchmarking Contemporary Deep Learning Hardware and Frameworks:A Survey of Qualitative Metrics

2019-07-05 · Wei Dai, Daniel Berleant

This paper surveys benchmarking principles, machine learning devices including GPUs, FPGAs, and ASICs, and deep learning software frameworks. It also reviews these technologies with respect to benchmarking from the persp…

Benchmarking BIG-bench Machine Learning Deep Learning

Efficient Deployment of Vision-Language Models on Mobile Devices: A Case Study on OnePlus 13R

2025-07-11 · Pablo Robin Guerrero, Yueyang Pan, Sanidhya Kashyap arxiv

Vision-Language Models (VLMs) offer promising capabilities for mobile devices, but their deployment faces significant challenges due to computational limitations and energy inefficiency, especially for real-time applicat…

A Study on Inference Latency for Vision Transformers on Mobile Devices

2025-10-29 · Zhuojin Li, Marco Paolieri, Leana Golubchik arxiv

Given the significant advances in machine learning techniques on mobile devices, particularly in the domain of computer vision, in this work we quantitatively study the performance characteristics of 190 real-world visio…

Robustness Analysis of Deep Learning Frameworks on Mobile Platforms

2021-09-20 · Amin Eslami Abyane, Hadi Hemmati

With the recent increase in the computational power of modern mobile devices, machine learning-based heavy tasks such as face detection and speech recognition are now integral parts of such devices. This requires framewo…

BIG-bench Machine Learning Deep Learning Face Detection Quantization+2

MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases

2024-06-12 · Rithesh Murthy, Liangwei Yang, Juntao Tan, Tulika Manoj Awalgaonkar 외

The deployment of Large Language Models (LLMs) and Large Multimodal Models (LMMs) on mobile devices has gained significant attention due to the benefits of enhanced privacy, stability, and personalization. However, the h…

Benchmarking Model Compression Quantization