Distribution Aware Metrics for Conditional Natural Language Generation
Traditional automated metrics for evaluating conditional natural language generation use pairwise comparisons between a single generated text and the best-matching gold-standard ground truth text. When multiple ground truths are available, scores are aggregated using an average or max operation across references. While this approach works well when diversity in the ground truth data (i.e. dispersion of the distribution of conditional texts) can be ascribed to noise, such as in automated speech recognition, it does not allow for robust evaluation in the case where diversity in the ground truths represents signal for the model. In this work we argue that existing metrics are not appropriate for domains such as visual description or summarization where ground truths are semantically diverse, and where the diversity in those captions captures useful additional information about the context. We propose a novel paradigm for multi-candidate evaluation of conditional language generation models, and a new family of metrics that compare the distributions of reference and model-generated caption sets using small sample sets of each. We demonstrate the utility of our approach with a case study in visual description: where we show that existing models optimize for single-description quality over diversity, and gain some insights into how sampling methods and temperature impact description quality and diversity.
Code (0)
등록된 구현이 없습니다.
Tasks
Diversityspeech-recognitionSpeech RecognitionText GenerationSimilar Papers 제목 키워드 기반
PolyAlign: Conditional Human-Distribution Alignment
Post-training methods such as supervised fine-tuning (SFT) and preference optimization typically align language models toward a single global assistant behavior. While effective for improving average helpfulness, this ca…
Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation
Token-level adaptive training approaches can alleviate the token imbalance problem and thus improve neural machine translation, through re-weighting the losses of different target tokens based on specific statistical met…
Language ModellingMachine TranslationSentenceTranslationReward Gaming in Conditional Text Generation
To align conditional text generation model outputs with desired behaviors, there has been an increasing focus on training the model using reinforcement learning (RL) with reward functions learned from human annotations. …
Conditional Text GenerationReinforcement Learning (RL)Text GenerationLabel Geometry Aware Discriminator for Conditional Generative Networks
Multi-domain image-to-image translation with conditional Generative Adversarial Networks (GANs) can generate highly photo realistic images with desired target classes, yet these synthetic images have not always been help…
Data Augmentationimage-classificationImage ClassificationImage-to-Image TranslationLogit-Coordinate Generative Models for Mixed Continuous-Categorical Tabular Data
Mixed continuous--categorical data pose a representation problem for continuous generative models. Flow Matching and Gaussian diffusion operate in Euclidean spaces, whereas categorical laws lie on probability simplices a…