Topicalization in Language Models: A Case Study on Japanese
Humans use different wordings depending on the context to facilitate efficient communication. For example, instead of completely new information, information related to the preceding context is typically placed at the sentence-initial position. In this study, we analyze whether neural language models (LMs) can capture such discourse-level preferences in text generation. Specifically, we focus on a particular aspect of discourse, namely the topic-comment structure. To analyze the linguistic knowledge of LMs separately, we chose the Japanese language, a topic-prominent language, for designing probing tasks, and we created human topicalization judgment data by crowdsourcing. Our experimental results suggest that LMs have different generalizations from humans; LMs exhibited less context-dependent behaviors toward topicalization judgment. These results highlight the need for the additional inductive biases to guide LMs to achieve successful discourse-level generalization.
Code (0)
등록된 구현이 없습니다.
Tasks
PositionSentenceText GenerationSimilar Papers 제목 키워드 기반
Language Models as an Alternative Evaluator of Word Order Hypotheses: A Case Study in Japanese
We examine a methodology using neural language models (LMs) for analyzing the word order of language. This LM-based method has the potential to overcome the difficulties existing methods face, such as the propagation of …
validControlling the Voice of a Sentence in Japanese-to-English Neural Machine Translation
In machine translation, we must consider the difference in expression between languages. For example, the active/passive voice may change in Japanese-English translation. The same verb in Japanese may be translated into …
DecoderMachine TranslationSentenceText Summarization+1Cost of Reasoning in non-English Languages: A Case Study on Japanese
Reasoning Language Models (RLMs) achieve their strongest performance when they reason in English, the language for which reasoning-oriented training data is most abundant. However, reasoning trace is a clue for model int…
On true empty category
According to Chomsky (1981, 1986), empty categories consist of PRO, pro, trace, and variable. However, some empty object positions seem to be incompatible with extant empty categories. Given this, Li (2007a, 2007b, 2014)…
PositionHow do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese
This paper investigates the effect of tokenizers on the downstream performance of pretrained language models (PLMs) in scriptio continua languages where no explicit spaces exist between words, using Japanese as a case st…