TensorFlow-Serving: Flexible, High-Performance ML Serving
We describe TensorFlow-Serving, a system to serve machine learning models inside Google which is also available in the cloud and via open-source. It is extremely flexible in terms of the types of ML platforms it supports, and ways to integrate with systems that convey new models and updated versions from training to serving. At the same time, the core code paths around model lookup and inference have been carefully optimized to avoid performance pitfalls observed in naive implementations. Google uses it in many production deployments, including a multi-tenant model hosting service called TFS^2.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningVocal Bursts Intensity PredictionSimilar Papers 제목 키워드 기반
On the Cost of Model-Serving Frameworks: An Experimental Evaluation
In machine learning (ML), the inference phase is the process of applying pre-trained models to new, unseen data with the objective of making predictions. During the inference phase, end-users interact with ML services to…
image-classificationImage ClassificationMalware DetectionSentiment Analysis+1FlexServe: Deployment of PyTorch Models as Flexible REST Endpoints
The integration of artificial intelligence capabilities into modern software systems is increasingly being simplified through the use of cloud-based machine learning services and representational state transfer architect…
Scaling TensorFlow to 300 million predictions per second
We present the process of transitioning machine learning models to the TensorFlow framework at a large scale in an online advertising ecosystem. In this talk we address the key challenges we faced and describe how we suc…
BIG-bench Machine LearningNVIDIA FLARE: Federated Learning from Simulation to Real-World
Federated learning (FL) enables building robust and generalizable AI models by leveraging diverse datasets from multiple collaborators without centralizing the data. We created NVIDIA FLARE as an open-source software dev…
Federated LearningPrivacy PreservingLightSeq: A High Performance Inference Library for Transformers
Transformer, BERT and their variants have achieved great success in natural language processing. Since Transformer models are huge in size, serving these models is a challenge for real industrial applications. In this pa…
GPUMachine TranslationTranslationVocal Bursts Intensity Prediction