Rethinking Streaming Machine Learning Evaluation
While most work on evaluating machine learning (ML) models focuses on computing accuracy on batches of data, tracking accuracy alone in a streaming setting (i.e., unbounded, timestamp-ordered datasets) fails to appropriately identify when models are performing unexpectedly. In this position paper, we discuss how the nature of streaming ML problems introduces new real-world challenges (e.g., delayed arrival of labels) and recommend additional metrics to assess streaming ML performance.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningPositionSimilar Papers 제목 키워드 기반
Streaming CTR Prediction: Rethinking Recommendation Task for Real-World Streaming Data
The Click-Through Rate (CTR) prediction task is critical in industrial recommender systems, where models are usually deployed on dynamic streaming data in practical applications. Such streaming data in real-world recomme…
Click-Through Rate PredictionPredictionRecommendation SystemsStream-level Latency Evaluation for Simultaneous Machine Translation
Simultaneous machine translation has recently gained traction thanks to significant quality improvements and the advent of streaming applications. Simultaneous translation systems need to find a trade-off between transla…
Machine TranslationSentenceTranslationDynamic Adaptation in Data Storage: Real-Time Machine Learning for Enhanced Prefetching
The exponential growth of data storage demands has necessitated the evolution of hierarchical storage management strategies [1]. This study explores the application of streaming machine learning [3] to revolutionize data…
Computational EfficiencyFeature EngineeringManagementDeep Learning with a Rethinking Structure for Multi-label Classification
Multi-label classification (MLC) is an important class of machine learning problems that come with a wide spectrum of applications, each demanding a possibly different evaluation criterion. When solving the MLC problems,…
ClassificationDeep LearningGeneral ClassificationMulti-Label Classification+1A review on Machine Learning based User-Centric Multimedia Streaming Techniques
The multimedia content and streaming are a major means of information exchange in the modern era and there is an increasing demand for such services. This coupled with the advancement of future wireless networks B5G/6G a…