Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference
On-device machine learning (ML) moves computation from the cloud to personal devices, protecting user privacy and enabling intelligent user experiences. However, fitting models on devices with limited resources presents a major technical challenge: practitioners need to optimize models and balance hardware metrics such as model size, latency, and power. To help practitioners create efficient ML models, we designed and developed Talaria: a model visualization and optimization system. Talaria enables practitioners to compile models to hardware, interactively visualize model statistics, and simulate optimizations to test the impact on inference metrics. Since its internal deployment two years ago, we have evaluated Talaria using three methodologies: (1) a log analysis highlighting its growth of 800+ practitioners submitting 3,600+ models; (2) a usability survey with 26 users assessing the utility of 20 Talaria features; and (3) a qualitative interview with the 7 most active users about their experience using Talaria.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Your Inference Request Will Become a Black Box: Confidential Inference for Cloud-based Large Language Models
The increasing reliance on cloud-hosted Large Language Models (LLMs) exposes sensitive client data, such as prompts and responses, to potential privacy breaches by service providers. Existing approaches fail to ensure pr…
Computational EfficiencyTalking with the Theorem Prover to Interactively Solve Natural Language Inference
Developing Bug-Free Machine Learning Systems With Formal Mathematics
Noisy data, non-convex objectives, model misspecification, and numerical instability can all cause undesired behaviors in machine learning systems. As a result, detecting actual implementation errors can be extremely dif…
BIG-bench Machine LearningOptimizing Latent Space Directions For GAN-based Local Image Editing
Generative Adversarial Network (GAN) based localized image editing can suffer from ambiguity between semantic attributes. We thus present a novel objective function to evaluate the locality of an image edit. By introduci…
DisentanglementGenerative Adversarial NetworkSynchronous Bidirectional Inference for Neural Sequence Generation
In sequence to sequence generation tasks (e.g. machine translation and abstractive summarization), inference is generally performed in a left-to-right manner to produce the result token by token. The neural approaches, s…
Abstractive Text SummarizationMachine TranslationTranslation