Rethinking and Refining the Distinct Metric
Distinct is a widely used automatic metric for evaluating the diversity of language generation tasks. However, we observe that the original approach to calculating distinct scores has evident biases that tend to add higher penalties to longer sequences. In this paper, we refine the calculation of distinct scores by re-scaling the number of distinct tokens based on its expectation. We provide both empirical and theoretical evidence to show that our method effectively removes the biases exhibited in the original distinct score. Further analyses also demonstrate that the refined score correlates better with human evaluations.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityText GenerationSimilar Papers 제목 키워드 기반
Rethinking and Refining the Distinct Metric
Distinct-$n$ score\cite{Li2016} is a widely used automatic metric for evaluating diversity in language generation tasks. However, we observed that the original approach for calculating distinct scores has evident biases …
DiversityText GenerationDetZero: Rethinking Offboard 3D Object Detection with Long-term Sequential Point Clouds
Existing offboard 3D detectors always follow a modular pipeline design to take advantage of unlimited sequential point clouds. We have found that the full potential of offboard 3D detectors is not explored mainly due to …
3D Multi-Object Tracking3D Object DetectionObjectobject-detection+1Rethinking the Reference-based Distinctive Image Captioning
Distinctive Image Captioning (DIC) -- generating distinctive captions that describe the unique details of a target image -- has received considerable attention over the last few years. A recent DIC work proposes to gener…
AttributeBenchmarkingImage CaptioningRethinking Layered Graphic Design Generation with a Top-Down Approach
Graphic design is crucial for conveying ideas and messages. Designers usually organize their work into objects, backgrounds, and vectorized text layers to simplify editing. However, this workflow demands considerable exp…
Hierarchical Interaction Networks with Rethinking Mechanism for Document-level Sentiment Analysis
Document-level Sentiment Analysis (DSA) is more challenging due to vague semantic links and complicate sentiment information. Recent works have been devoted to leveraging text summarization and have achieved promising re…
Sentiment AnalysisSentiment ClassificationText Classification