uBLEU: Uncertainty-Aware Automatic Evaluation Method for Open-Domain Dialogue Systems
Because open-domain dialogues allow diverse responses, basic reference-based metrics such as BLEU do not work well unless we prepare a massive reference set of high-quality responses for input utterances. To reduce this burden, a human-aided, uncertainty-aware metric, ΔBLEU, has been proposed; it embeds human judgment on the quality of reference outputs into the computation of multiple-reference BLEU. In this study, we instead propose a fully automatic, uncertainty-aware evaluation method for open-domain dialogue systems, υBLEU. This method first collects diverse reference responses from massive dialogue data and then annotates their quality judgments by using a neural network trained on automatically collected training data. Experimental results on massive Twitter data confirmed that υBLEU is comparable to ΔBLEU in terms of its correlation with human judgment and that the state of the art automatic evaluation method, RUBER, is improved by integrating υBLEU.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
DoubleU-Net: A Deep Convolutional Neural Network for Medical Image Segmentation
Semantic image segmentation is the process of labeling each pixel of an image with its corresponding class. An encoder-decoder based approach, like U-Net and its variants, is a popular strategy for solving medical image …
Cell SegmentationColorectal Polyps CharacterizationImage SegmentationLesion Segmentation+5DoubleU-Net++: Architecture with Exploit Multiscale Features for Vertebrae Segmentation
Accurate segmentation of the vertebra is an important prerequisite in various medical applications (E.g. tele surgery) to assist surgeons. Following the successful development of deep neural networks, recent studies have…
SegmentationDoubleU-NetPlus: A Novel Attention and Context Guided Dual U-Net with Multi-Scale Residual Feature Fusion Network for Semantic Segmentation of Medical Images
Accurate segmentation of the region of interest in medical images can provide an essential pathway for devising effective treatment plans for life-threatening diseases. It is still challenging for U-Net, and its state-of…
Image SegmentationMedical Image SegmentationSemantic SegmentationUncertainty-Aware Multiple Instance Learning from Large-Scale Long Time Series Data
We propose a novel framework to classify large-scale time series data with long duration. Long time seriesclassification (L-TSC) is a challenging problem because the dataoften contains a large amount of irrelevant inform…
Multiple Instance LearningTime SeriesTime Series AnalysisPerception Score, A Learned Metric for Open-ended Text Generation Evaluation
Automatic evaluation for open-ended natural language generation tasks remains a challenge. Existing metrics such as BLEU show a low correlation with human judgment. We propose a novel and powerful learning-based evaluati…
Text Generation