Native Language Identification with Large Language Models

2023-12-13 · Wei zhang, Alexandre Salle

We present the first experiments on Native Language Identification (NLI) using LLMs such as GPT-4. NLI is the task of predicting a writer's first language by analyzing their writings in a second language, and is used in second language acquisition and forensic linguistics. Our results show that GPT models are proficient at NLI classification, with GPT-4 setting a new performance record of 91.7% on the benchmark TOEFL11 test set in a zero-shot setting. We also show that unlike previous fully-supervised settings, LLMs can perform NLI without being limited to a set of known classes, which has practical implications for real-world applications. Finally, we also show that LLMs can provide justification for their choices, providing reasoning based on spelling errors, syntactic patterns, and usage of directly translated linguistic patterns.

📄 PDF Abstract BibTeX arXiv:2312.07819

Code (0)

등록된 구현이 없습니다.

Tasks

Language Acquisition Language Identification Native Language Identification

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Position-Wise Feed-Forward Layer 설명 없음

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Transformer A Transformer is a model architecture that eschews recurrence and instead relies entirely on an [attention…

Improving Language Identification of Accented Speech

2022-03-31 · Kunnar Kukk, Tanel Alumäe

Language identification from speech is a common preprocessing step in many spoken language processing systems. In recent years, this field has seen fast progress, mostly due to the use of self-supervised models pretraine…

Language Identification speech-recognition Speech Recognition Spoken language identification

Native Language Identification Using Large, Longitudinal Data

2014-05-01 · LREC 2014 5 · Xiao Jiang, Yufan Guo, Jeroen Geertzen, Dora Alexopoulou 외

Native Language Identification (NLI) is a task aimed at determining the native language (L1) of learners of second language (L2) on the basis of their written texts. To date, research on NLI has focused on relatively sma…

BIG-bench Machine Learning Language Identification Native Language Identification Text Classification

Native Language Identification Using a Mixture of Character and Word N-grams

2017-09-01 · WS 2017 9 · Elham Mohammadi, Hadi Veisi, Hessam Amini

Native language identification (NLI) is the task of determining an author{'}s native language, based on a piece of his/her writing in a second language. In recent years, NLI has received much attention due to its challen…

Language Acquisition Language Identification Native Language Identification

Measuring Interlanguage: Native Language Identification with L1-influence Metrics

2012-05-01 · LREC 2012 5 · Julian Brooke, Graeme Hirst

The task of native language (L1) identification suffers from a relative paucity of useful training corpora, and standard within-corpus evaluation is often problematic due to topic bias. In this paper, we introduce a meth…

Language Acquisition Language Identification Machine Translation Native Language Identification+3

On the Development of a Large Scale Corpus for Native Language Identification

2018-12-10 · TLT17 2018 12 · Thomas Hudson, Sardar Jaf

Native Language Identification (NLI) is the task of identifying an author’s native language from their writings in a second language. In this paper, we introduce a new corpus (italki), which is larger than the current co…

BIG-bench Machine Learning Language Identification Native Language Identification