Evaluation Metrics of Language Generation Models for Synthetic Traffic Generation Tasks
Many Natural Language Generation (NLG) tasks aim to generate a single output text given an input prompt. Other settings require the generation of multiple texts, e.g., for Synthetic Traffic Generation (STG). This generation task is crucial for training and evaluating QA systems as well as conversational agents, where the goal is to generate multiple questions or utterances resembling the linguistic variability of real users. In this paper, we show that common NLG metrics, like BLEU, are not suitable for evaluating STG. We propose and evaluate several metrics designed to compare the generated traffic to the distribution of real user texts. We validate our metrics with an automatic procedure to verify whether they capture different types of quality issues of generated data; we also run human annotations to verify the correlation with human judgements. Experiments on three tasks, i.e., Shopping Utterance Generation, Product Question Generation and Query Auto Completion, demonstrate that our metrics are effective for evaluating STG tasks, and improve the agreement with human judgement up to 20% with respect to common NLG metrics. We believe these findings can pave the way towards better solutions for estimating the representativeness of synthetic text data.
Code (0)
등록된 구현이 없습니다.
Tasks
Question GenerationQuestion-GenerationText GenerationSimilar Papers 제목 키워드 기반
Synthetic Traffic Generation with Wasserstein Generative Adversarial Networks
Network traffic data are critical for network research. With the help of synthetic traffic, researchers can readily generate data for network simulation and performance evaluation. However, the state-of-the-art traffic g…
Intelligent CommunicationSynthetic Data GenerationQuantifying the Privacy Implications of High-Fidelity Synthetic Network Traffic
To address the scarcity and privacy concerns of network traffic data, various generative models have been developed to produce synthetic traffic. However, synthetic traffic is not inherently privacy-preserving, and the e…
Lightweight GenAI for Network Traffic Synthesis: Fidelity, Augmentation, and Classification
Accurate Network Traffic Classification (NTC) is increasingly constrained by limited labeled data and strict privacy requirements. While Network Traffic Generation (NTG) provides an effective means to mitigate data scarc…
Computational EfficiencyData AugmentationSynthetic flow-based cryptomining attack generation through Generative Adversarial Networks
Due to the growing rise of cyber attacks in the Internet, flow-based data sets are crucial to increase the performance of the Machine Learning (ML) components that run in network-based intrusion detection systems (IDS). …
Data AugmentationIntrusion DetectionA Graph-based Methodology for the Sensorless Estimation of Road Traffic Profiles
Traffic forecasting models rely on data that needs to be sensed, processed, and stored. This requires the deployment and maintenance of traffic sensing infrastructure, often leading to unaffordable monetary costs. The la…