paper-with-me

홈 › Papers

Scaling, Simplification, and Adaptation: Lessons from Pretraining on Machine-Translated Text

2025-09-22 · Dan John Velasco, Matthew Theodore Roque arxiv

Most languages lack sufficient data for large-scale monolingual pretraining, creating a "data wall." Multilingual pretraining helps but is limited by language imbalance and the "curse of multilinguality." An alternative is to translate high-resource text with machine translation (MT), which raises three questions: (1) How does MT-derived data scale with model capacity? (2) Can source-side transformations (e.g., simplifying English with an LLM) improve generalization to native text? (3) How well do models pretrained on MT-derived data adapt when continually trained on limited native text? We investigate these questions by translating English into Indonesian and Tamil--two typologically distant, lower-resource languages--and pretraining GPT-2 models (124M-774M) on native or MT-derived corpora from raw and LLM-simplified English. We evaluate cross-entropy loss on native text, along with accuracy on syntactic probes and downstream tasks. Our results show that (1) MT-pretrained models benefit from scaling; (2) source-side simplification harms generalization to native text; and (3) adapting MT-pretrained models on native text often yields better performance than native-only models, even with less native data. However, tasks requiring cultural nuance (e.g., toxicity detection) demand more exposure to native data.

📄 PDF Abstract BibTeX arXiv:2509.17317

Code (0)

등록된 구현이 없습니다.

Tasks

Machine Translation

Similar Papers 제목 키워드 기반

ARTIST: ARTificial Intelligence for Simplified Text

2023-08-25 · Lorenzo Corti, Jie Yang

Complex text is a major barrier for many citizens when accessing public information and knowledge. While often done manually, Text Simplification is a key Natural Language Processing task that aims for reducing the lingu…

Text Simplification

Optimizing Statistical Machine Translation for Text Simplification

2016-01-01 · TACL 2016 1 · Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen 외

Most recent sentence simplification systems use basic machine translation models to learn lexical and syntactic paraphrases from a manually simplified parallel corpus. These methods are limited by the quality and quantit…

Machine TranslationSentenceText SimplificationTranslation

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining

2025-08-14 · DatologyAI, :, Pratyush Maini, Vineeth Dorna 외 arxiv

Recent advances in large language model (LLM) pretraining have shown that simply scaling data quantity eventually leads to diminishing returns, hitting a data wall. In response, the use of synthetic data for pretraining …

Synthetic Data Generation

Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve?

2024-10-08 · Fırat Öncel, Matthias Bethge, Beyza Ermis, Mirco Ravanelli 외

In the last decade, the generalization and adaptation abilities of deep learning models were typically evaluated on fixed training and test distributions. Contrary to traditional deep learning, large language models (LLM…

Deep Learning

Asymmetric Conflict and Synergy in Post-training for LLM-based Multilingual Machine Translation

2025-02-16 · Tong Zheng, Yan Wen, Huiwen Bao, Junfeng Guo 외

The emergence of Large Language Models (LLMs) has advanced the multilingual machine translation (MMT), yet the Curse of Multilinguality (CoM) remains a major challenge. Existing work in LLM-based MMT typically mitigates …

Machine Translation