Sailor: Open Language Models for South-East Asia

2024-04-04 · Longxu Dou, Qian Liu, Guangtao Zeng, Jia Guo, Jiahui Zhou, Wei Lu, Min Lin

We present Sailor, a family of open language models ranging from 0.5B to 7B parameters, tailored for South-East Asian (SEA) languages. These models are continually pre-trained from Qwen1.5, a great language model for multilingual use cases. From Qwen1.5, Sailor models accept 200B to 400B tokens, primarily covering the languages of English, Chinese, Vietnamese, Thai, Indonesian, Malay, and Lao. The training leverages several techniques, including BPE dropout for improving the model robustness, aggressive data cleaning and deduplication, and small proxy models to optimize data mixture. Experimental results on four typical tasks indicate that Sailor models demonstrate strong performance across different benchmarks, including commonsense reasoning, question answering, reading comprehension and examination. Embracing the open-source spirit, we share our insights through this report to spark a wider interest in developing large language models for multilingual use cases.

📄 PDF Abstract BibTeX arXiv:2404.03608

Code (3)

epfllm/megatron-llm 공식 구현 pytorch

sail-sg/sailor-llm 공식 구현 pytorch

hit-scir/huozi pytorch

Tasks

Language ModelingLanguage Modelling Question Answering Reading Comprehension

Methods 이 논문이 사용한 방법론

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

SEA-LION-Embedding: Open and Reproducible Text Embeddings for Southeast Asia

2026-06-02 · Peerat Limkonchotiwat, Raymond Ng, Sarana Nutanong, Jian Gang Ngui arxiv

Text embeddings are fundamental to many downstream applications, making robustness important for real-world NLP. However, most recent state-of-the-art embedding models are not reproducible because they rely on closed or …

Dolphin: A Large-Scale Automatic Speech Recognition Model for Eastern Languages

2025-03-26 · Yangyang Meng, Jinpeng Li, Guodong Lin, Yu Pu 외

This report introduces Dolphin, a large-scale multilingual automatic speech recognition (ASR) model that extends the Whisper architecture to support a wider range of languages. Our approach integrates in-house proprietar…

Automatic Speech Recognition Automatic Speech Recognition (ASR)speech-recognition Speech Recognition

OpenSeal: Good, Fast, and Cheap Construction of an Open-Source Southeast Asian LLM via Parallel Data

2026-02-02 · Tan Sang Nguyen, Muhammad Reza Qorib, Hwee Tou Ng arxiv

Large language models (LLMs) have proven to be effective tools for a wide range of natural language processing (NLP) applications. Although many LLMs are multilingual, most remain English-centric and perform poorly on lo…

Continual Pretraining