홈 › Papers

PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency

2024-10-10 · Preferred Elements, :, Kenshin Abe, Kaizaburo Chubachi, Yasuhiro Fujita, Yuta Hirokawa, Kentaro Imajo, Toshiki Kataoka, Hiroyoshi Komatsu, Hiroaki Mikami, Tsuguo Mogami, Shogo Murai, Kosuke Nakago, Daisuke Nishino, Toru Ogawa, Daisuke Okanohara, Yoshihiko Ozaki, Shotaro Sano, Shuji Suzuki, Tianqi Xu, Toshihiko Yanase

We introduce PLaMo-100B, a large-scale language model designed for Japanese proficiency. The model was trained from scratch using 2 trillion tokens, with architecture such as QK Normalization and Z-Loss to ensure training stability during the training process. Post-training techniques, including Supervised Fine-Tuning and Direct Preference Optimization, were applied to refine the model's performance. Benchmark evaluations suggest that PLaMo-100B performs well, particularly in Japanese-specific tasks, achieving results that are competitive with frontier models like GPT-4. The base model is available at https://huggingface.co/pfnet/plamo-100b.

📄 PDF Abstract BibTeX arXiv:2410.07563

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage Modelling

Methods 이 논문이 사용한 방법론

BASE 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Residual Connection 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Adam 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

PLaMo 2.1-VL Technical Report

2026-04-21 · Tommi Kerola, Yuya Masuda, Takashi Masuko, Toshiki Nakanishi 외 arxiv

We introduce PLaMo 2.1-VL, a lightweight Vision Language Model (VLM) for autonomous devices, available in 8B and 2B variants and designed for local and edge deployment with Japanese-language operation. Focusing on Visual…

Synthetic Data Generation Visual Question Answering Anomaly Detection Visual Grounding

PLaMo 2 Technical Report

2025-09-05 · Preferred Networks, :, Kaizaburo Chubachi, Yasuhiro Fujita 외 arxiv

In this report, we introduce PLaMo 2, a series of Japanese-focused large language models featuring a hybrid Samba-based architecture that transitions to full attention via continual pre-training to support 32K token cont…

Computational Efficiency

Enhancing Large Language Model-based Speech Recognition by Contextualization for Rare and Ambiguous Words

2024-08-15 · Kento Nozawa, Takashi Masuko, Toru Taniguchi

We develop a large language model (LLM) based automatic speech recognition (ASR) system that can be contextualized by providing keywords as prior information in text prompts. We adopt decoder-only architecture and use ou…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Decoder Language Modeling+4

Item Development and Scoring for Japanese Oral Proficiency Testing

2012-05-01 · LREC 2012 5 · Hitokazu Matsushita, Deryle Lonsdale

This study introduces and evaluates a computerized approach to measuring Japanese L2 oral proficiency. We present a testing and scoring method that uses a type of structured speech called elicited imitation (EI) to evalu…

Language Modeling Language Modelling Speech Recognition

Measuring Diversified Proficiency of Japanese Learners of English

2016-10-01 · PACLIC 2016 10 · Yasunari Harada

Speech Recognition