WMT20 Document-Level Markable Error Exploration
Even though sentence-centric metrics are used widely in machine translation evaluation, document-level performance is at least equally important for professional usage. In this paper, we bring attention to detailed document-level evaluation focused on markables (expressions bearing most of the document meaning) and the negative impact of various markable error phenomena on the translation. For an annotation experiment of two phases, we chose Czech and English documents translated by systems submitted to WMT20 News Translation Task. These documents are from the News, Audit and Lease domains. We show that the quality and also the kind of errors varies significantly among the domains. This systematic variance is in contrast to the automatic evaluation results. We inspect which specific markables are problematic for MT systems and conclude with an analysis of the effect of markable error types on the MT performance measured by humans and automatic evaluation tools.
Code (1)
Tasks
Machine TranslationSentenceTranslationSimilar Papers 제목 키워드 기반
Revise: A Framework for Revising OCRed text in Practical Information Systems with Data Contamination Strategy
Recent advances in Large Language Models (LLMs) have significantly improved the field of Document AI, demonstrating remarkable performance on document understanding tasks such as question answering. However, existing app…
Synthetic Data GenerationQuestion AnsweringDocument AIGraphy'our Data: Towards End-to-End Modeling, Exploring and Generating Report from Raw Data
Large Language Models (LLMs) have recently demonstrated remarkable performance in tasks such as Retrieval-Augmented Generation (RAG) and autonomous AI agent workflows. Yet, when faced with large sets of unstructured docu…
AI AgentRAGRetrieval-augmented GenerationSurveyIs ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation
ChatGPT, a large-scale language model based on the advanced GPT-3.5 architecture, has shown remarkable potential in various Natural Language Processing (NLP) tasks. However, there is currently a dearth of comprehensive s…
Grammatical Error CorrectionIn-Context LearningLanguage ModelingLanguage Modelling+1Document-level grammatical error correction
Document-level context can provide valuable information in grammatical error correction (GEC), which is crucial for correcting certain errors and resolving inconsistencies. In this paper, we investigate context-aware app…
Grammatical Error CorrectionNMTSentenceText Network Exploration via Heterogeneous Web of Topics
A text network refers to a data type that each vertex is associated with a text document and the relationship between documents is represented by edges. The proliferation of text networks such as hyperlinked webpages and…