PMLM

2000년 도입 · 논문 3편에서 사용

Probabilistically Masked Language Model, or PMLM, is a type of language model that utilizes a probabilistic masking scheme, aiming to bridge the gap between masked and autoregressive language models. The basic idea behind the connection of two categories of models is similar to MADE by Germain et al (2015). PMLM is a masked language model with a probabilistic masking scheme, which defines the way sequences are masked by following a probabilistic distribution. The authors employ a simple uniform distribution of the masking ratio and name the model as u-PMLM.

출처: Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order

소개 논문: Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order