Evaluating Transformer Models and Human Behaviors on Chinese Character Naming
Neural network models have been proposed to explain the grapheme-phoneme mapping process in humans for many alphabet languages. These models not only successfully learned the correspondence of the letter strings and their pronunciation, but also captured human behavior in nonce word naming tasks. How would the neural models perform for a non-alphabet language (e.g., Chinese) unknown character task? How well would the model capture human behavior? In this study, we first collect human speakers' answers on unknown character naming tasks and then evaluate a set of transformer models by comparing their performances with human behaviors on an unknown Chinese character naming task. We found that the models and humans behaved very similarly, that they had similar accuracy distribution for each character, and had a substantial overlap in answers. In addition, the models' answers are highly correlated with humans' answers. These results suggested that the transformer models can well capture human's character naming behavior.
Code (1)
Similar Papers 제목 키워드 기반
Towards Evaluating the Robustness of Chinese BERT Classifiers
Recent advances in large-scale language representation models such as BERT have improved the state-of-the-art performances in many NLP tasks. Meanwhile, character-level Chinese NLP models, including BERT for Chinese, hav…
Chinese WPLC: A Chinese Dataset for Evaluating Pretrained Language Models on Word Prediction Given Long-Range Context
This paper presents a Chinese dataset for evaluating pretrained language models on Word Prediction given Long-term Context (Chinese WPLC). We propose both automatic and manual selection strategies tailored to Chinese to …
DiversityLanguage ModelingLanguage ModellingMECT: Multi-Metadata Embedding based Cross-Transformer for Chinese Named Entity Recognition
Recently, word enhancement has become very popular for Chinese Named Entity Recognition (NER), reducing segmentation errors and increasing the semantic and boundary information of Chinese words. However, these methods te…
BenchmarkingChinese Named Entity Recognitionnamed-entity-recognitionNamed Entity Recognition+2How Far Are LLMs from Believable AI? A Benchmark for Evaluating the Believability of Human Behavior Simulation
In recent years, AI has demonstrated remarkable capabilities in simulating human behaviors, particularly those implemented with large language models (LLMs). However, due to the lack of systematic evaluation of LLMs' sim…
AI AgentLanguage ModellingLarge Language ModelAutomatic Assessment of Divergent Thinking in Chinese Language with TransDis: A Transformer-Based Language Model Approach
Language models have been increasingly popular for automatic creativity assessment, generating semantic distances to objectively measure the quality of creative ideas. However, there is currently a lack of an automatic a…
Language ModelingLanguage Modellingvalid