Poro 34B and the Blessing of Multilinguality
The pretraining of state-of-the-art large language models now requires trillions of words of text, which is orders of magnitude more than available for the vast majority of languages. While including text in more than one language is an obvious way to acquire more pretraining data, multilinguality is often seen as a curse, and most model training efforts continue to focus near-exclusively on individual large languages. We believe that multilinguality can be a blessing and that it should be possible to substantially improve over the capabilities of monolingual models for small languages through multilingual training. In this study, we introduce Poro 34B, a 34 billion parameter model trained for 1 trillion tokens of Finnish, English, and programming languages, and demonstrate that a multilingual training approach can produce a model that not only substantially advances over the capabilities of existing models for Finnish, but also excels in translation and is competitive in its class in generating English and programming languages. We release the model parameters, scripts, and data under open licenses at https://huggingface.co/LumiOpen/Poro-34B.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learning
While multilingual large language models generally perform adequately, and sometimes even rival English performance on high-resource languages (HRLs), they often significantly underperform on low-resource languages (LRLs…
Cross-Lingual TransferIn-Context LearningTowards Attribute-Entangled Controllable Text Generation: A Pilot Study of Blessing Generation
Controllable Text Generation (CTG) has obtained great success due to its fine-grained generation ability obtained by focusing on multiple attributes. However, most existing CTG researches overlook how to utilize the attr…
AttributeDiversityText GenerationLet Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals
Cardiac signals, such as the electrocardiogram, convey a significant amount of information about the health status of a patient which is typically summarized by a clinician in the form of a clinical report, a cumbersome …
The Blessings of Multiple Causes: A Reply to Ogburn et al. (2019)
Ogburn et al. (2019, arXiv:1910.05438) discuss "The Blessings of Multiple Causes" (Wang and Blei, 2018, arXiv:1805.06826). Many of their remarks are interesting. But they also claim that the paper has "foundational error…
Multiscale porosity in mesoporous bioglass 3D-printed scaffolds for bone regeneration
In order to increase the bone forming ability of MBG-PCL composite scaffold, microporosity was created in the struts of 3D-printed MBG-PCL scaffolds for the manufacturing of a construct with a multiscale porosity consist…