Comparing Character-level Neural Language Models Using a Lexical Decision Task
What is the information captured by neural network models of language? We address this question in the case of character-level recurrent neural language models. These models do not have explicit word representations; do they acquire implicit ones? We assess the lexical capacity of a network using the lexical decision task common in psycholinguistics: the system is required to decide whether or not a string of characters forms a word. We explore how accuracy on this task is affected by the architecture of the network, focusing on cell type (LSTM vs. SRN), depth and width. We also compare these architectural properties to a simple count of the parameters of the network. The overall number of parameters in the network turns out to be the most important predictor of accuracy; in particular, there is little evidence that deeper networks are beneficial for this task.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Humans and transformer LMs: Abstraction drives language learning
Categorization is a core component of human linguistic competence. We investigate how a transformer-based language model (LM) learns linguistic categories by comparing its behaviour over the course of training to behavio…
Language AcquisitionUNBNLP at SemEval-2021 Task 1: Predicting lexical complexity with masked language models and character-level encoders
In this paper, we present three supervised systems for English lexical complexity prediction of single and multiword expressions for SemEval-2021 Task 1. We explore the use of statistical baseline features, masked langua…
Lexical Complexity PredictionPredictionSyntax-Aware Language Modeling with Recurrent Neural Networks
Neural language models (LMs) are typically trained using only lexical features, such as surface forms of words. In this paper, we argue this deprives the LM of crucial syntactic signals that can be detected at high confi…
Language ModelingLanguage ModellingOn the Distribution of Lexical Features at Multiple Levels of Analysis
Natural language processing has increasingly moved from modeling documents and words toward studying the people behind the language. This move to working with data at the user or community level has presented the field w…
Document ClassificationSentiment AnalysisShape vs. Context: Examining Human--AI Gaps in Ambiguous Japanese Character Recognition
High text recognition performance does not guarantee that Vision-Language Models (VLMs) share human-like decision patterns when resolving ambiguity. We investigate this behavioral gap by directly comparing humans and VLM…