MTG: A Benchmarking Suite for Multilingual Text Generation
We introduce MTG, a new benchmark suite for training and evaluating multilingual text generation. It is the first and largest multilingual multiway text generation benchmark with 400k human-annotated data for four tasks (story generation, question generation, title generation and text summarization) across five languages (English, German, French, Spanish and Chinese). Its multiway characteristic makes it possible to create cross-lingual data between any of two languages and thus boosts the direct cross-lingual knowledge transfer. Based on it, we set various evaluation scenarios and make a deep analysis of several popular multilingual generation models from different aspects. Our benchmark suite will encourage multilingualism for the text generation community with more human-annotated parallel data and more diverse generation scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingQuestion GenerationQuestion-GenerationStory GenerationText GenerationText SummarizationTransfer LearningSimilar Papers 제목 키워드 기반
MTG: A Benchmark Suite for Multilingual Text Generation
We introduce MTG, a new benchmark suite for training and evaluating multilingual text generation. It is the first-proposed multilingual multiway text generation dataset with the largest human-annotated data (400k). It in…
Question GenerationQuestion-GenerationStory GenerationText Generation+2One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support
Recent strides in Large Language Models (LLMs) have saturated many Natural Language Processing (NLP) benchmarks, emphasizing the need for more challenging ones to properly assess LLM capabilities. However, domain-specifi…
BenchmarkingInformation RetrievalLanguage ModellingLegal Reasoning+2IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages
Natural Language Generation (NLG) for non-English languages is hampered by the scarcity of datasets in these languages. In this paper, we present the IndicNLG Benchmark, a collection of datasets for benchmarking NLG for …
ArticlesBenchmarkingHeadline GenerationMachine Translation+6Benchmarking LLM Guardrails in Handling Multilingual Toxicity
With the ubiquity of Large Language Models (LLMs), guardrails have become crucial to detect and defend against toxic content. However, with the increasing pervasiveness of LLMs in multilingual scenarios, their effectiven…
BenchmarkingMultilingual Conceptual Coverage in Text-to-Image Models
We propose "Conceptual Coverage Across Languages" (CoCo-CroLa), a technique for benchmarking the degree to which any generative text-to-image system provides multilingual parity to its training language in terms of tangi…
Benchmarking