MLMLM: Link Prediction with Mean Likelihood Masked Language Model

2020-09-15 · Findings (ACL) 2021 8 · Louis Clouatre, Philippe Trempe, Amal Zouaq, Sarath Chandar

Knowledge Bases (KBs) are easy to query, verifiable, and interpretable. They however scale with man-hours and high-quality data. Masked Language Models (MLMs), such as BERT, scale with computing power as well as unstructured raw text data. The knowledge contained within those models is however not directly interpretable. We propose to perform link prediction with MLMs to address both the KBs scalability issues and the MLMs interpretability issues. To do that we introduce MLMLM, Mean Likelihood Masked Language Model, an approach comparing the mean likelihood of generating the different entities to perform link prediction in a tractable manner. We obtain State of the Art (SotA) results on the WN18RR dataset and the best non-entity-embedding based results on the FB15k-237 dataset. We also obtain convincing results on link prediction on previously unseen entities, making MLMLM a suitable approach to introducing new entities to a KB.

📄 PDF Abstract BibTeX arXiv:2009.07058

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage Modelling Link Prediction Prediction

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Interpretability 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Weight Decay 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

An Empirical Study on Pseudo-log-likelihood Bias Measures for Masked Language Models Using Paraphrased Sentences

2022-07-01 · NAACL (TrustNLP) 2022 7 · Bum Chul Kwon, Nandana Mihindukulasooriya

In this paper, we conduct an empirical study on a bias measure, log-likelihood Masked Language Model (MLM) scoring, on a benchmark dataset. Previous work evaluates whether MLMs are biased or not for certain protected att…

Language Modeling Language Modelling Sentence

Heteroscedastic Neural Networks for Path Loss Prediction with Link-Specific Uncertainty

2025-11-28 · Jonathan Ethier arxiv

Traditional and modern machine learning-based path loss models typically assume a constant prediction variance. We propose a neural network that jointly predicts the mean and link-specific variance by minimizing a Gaussi…

A Bayesian Tensor Factorization Model via Variational Inference for Link Prediction

2014-09-29 · Beyza Ermis, A. Taylan Cemgil

Probabilistic approaches for tensor factorization aim to extract meaningful structure from incomplete data by postulating low rank constraints. Recently, variational Bayesian (VB) inference techniques have successfully b…

Bayesian Inference Link Prediction Prediction Variational Inference

Information-Theoretic Discrete Diffusion

2025-10-28 · Moongyu Jeon, Sangwoo Shin, Dongjae Jeon, Albert No arxiv

We present an information-theoretic framework for discrete diffusion models that yields principled estimators of log-likelihood using score-matching losses. Inspired by the I-MMSE identity for the Gaussian setup, we deri…

Unmasking the Mask -- Evaluating Social Biases in Masked Language Models

2021-04-15 · Masahiro Kaneko, Danushka Bollegala

Masked Language Models (MLMs) have shown superior performances in numerous downstream NLP tasks when used as text encoders. Unfortunately, MLMs also demonstrate significantly worrying levels of social biases. We show tha…

Selection bias Sentence