PrOnto: Language Model Evaluations for 859 Languages
Evaluation datasets are critical resources for measuring the quality of pretrained language models. However, due to the high cost of dataset annotation, these resources are scarce for most languages other than English, making it difficult to assess the quality of language models. In this work, we present a new method for evaluation dataset construction which enables any language with a New Testament translation to receive a suite of evaluation datasets suitable for pretrained language model evaluation. The method critically involves aligning verses with those in the New Testament portion of English OntoNotes, and then projecting annotations from English to the target language, with no manual annotation required. We apply this method to 1051 New Testament translations in 859 and make them publicly available. Additionally, we conduct experiments which demonstrate the efficacy of our method for creating evaluation tasks which can assess language model quality.
Code (1)
Tasks
Language Model EvaluationLanguage ModelingLanguage ModellingmodelSimilar Papers 제목 키워드 기반
PRONTO: Preamble Overhead Reduction with Neural Networks for Coarse Synchronization
In IEEE 802.11 WiFi-based waveforms, the receiver performs coarse time and frequency synchronization using the first field of the preamble known as the legacy short training field (L-STF). The L-STF occupies upto 40% of …
CPUData AugmentationGPUDATA-DRIVEN PRONTO: a Model-free Solution for Numerical Optimal Control
This article addresses the problem of data-driven numerical optimal control for unknown nonlinear systems. In our scenario, we suppose to have the possibility of performing multiple experiments (or simulations) on the sy…
Draft-and-Prune: Improving the Reliability of Auto-formalization for Logical Reasoning
Auto-formalization (AF) translates natural-language reasoning problems into solver-executable programs, enabling symbolic solvers to perform sound logical deduction. In practice, however, AF pipelines are currently britt…
Logical ReasoningNon-Interactive Symbolic-Aided Chain-of-Thought for Logical Reasoning
This work introduces Symbolic-Aided Chain-of-Thought (CoT), an improved approach to standard CoT, for logical reasoning in large language models (LLMs). The key idea is to integrate lightweight symbolic representations i…
Logical ReasoningConcise and Organized Perception Facilitates Reasoning in Large Language Models
Exploiting large language models (LLMs) to tackle reasoning has garnered growing attention. It still remains highly challenging to achieve satisfactory results in complex logical problems, characterized by plenty of prem…
LAMBADAMath