A crowdsourcing approach to video quality assessment
The gold standard for measuring video quality is the subjective test, and the most prevalent is the ITU-T Rec. P.910, a lab-based subjective standard in use for the past two decades. However, in practice using ITU-T Rec. P.910 is slow, expensive, and requires a lab, which all create barriers to usage. As a result, most research papers that need to measure video quality don't use ITU-T Rec. P.910 but rather metrics that are not well correlated to subjective opinion. We provide an open-source extension of ITU-T Rec. P.910 based on crowdsourcing principles which address the speed, usage cost, and barrier to usage issues. We implement Absolute Category Rating (ACR), ACR with hidden reference (ACR-HR), Degradation Category Rating (DCR), and Comparison Category Rating (CCR). It includes rater, environment, hardware, and network qualifications, as well as gold and trapping questions to ensure quality. We have validated that the implementation is both accurate and highly reproducible compared to existing ITU-T Rec. P.910 lab studies.
Code (1)
Tasks
Video Quality AssessmentSimilar Papers 제목 키워드 기반
Ensuring Reliable Participation in Subjective Video Quality Tests Across Platforms
Subjective video quality assessment (VQA) is the gold standard for measuring end-user experience across communication, streaming, and UGC pipelines. Beyond high-validity lab studies, crowdsourcing offers accurate, reliab…
Video Quality AssessmentEvaluation of Automatic Video Captioning Using Direct Assessment
We present Direct Assessment, a method for manually assessing the quality of automatically-generated captions for video. Evaluating the accuracy of video captions is particularly difficult because for any given video cli…
Caption GenerationMachine TranslationTranslationVideo CaptioningLarge-Scale Study of Perceptual Video Quality
The great variations of videographic skills, camera designs, compression and processing protocols, and displays lead to an enormous variety of video impairments. Current no-reference (NR) video quality models are unable …
Video Quality AssessmentCrowdMOT: Crowdsourcing Strategies for Tracking Multiple Objects in Videos
Crowdsourcing is a valuable approach for tracking objects in videos in a more scalable manner than possible with domain experts. However, existing frameworks do not produce high quality results with non-expert crowdworke…
DiversityMulti-dimensional Speech Quality Assessment in Crowdsourcing
Subjective speech quality assessment is the gold standard for evaluating speech enhancement processing and telecommunication systems. The commonly used standard ITU-T Rec. P.800 defines how to measure speech quality in l…
Speech Enhancement