Papers Language Identification
“Language Identification” 태그가 달린 논문 852편 · 필터 해제
Can adversarial attacks by large language models be attributed?
Attributing outputs from Large Language Models (LLMs) in adversarial settings-such as cyberattacks and disinformation-presents significant challenges that are likely to grow in importance. We investigate this attribution…
AttributeLanguage IdentificationPrompt Engineering Using GPT for Word-Level Code-Mixed Language Identification in Low-Resource Dravidian Languages
Language Identification (LI) is crucial for various natural language processing tasks, serving as a foundational step in applications such as sentiment analysis, machine translation, and information retrieval. In multili…
Information RetrievalLanguage IdentificationMachine TranslationPrompt Engineering+1GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages
The need for large text corpora has increased with the advent of pretrained language models and, in particular, the discovery of scaling laws for these models. Most available corpora have sufficient data only for languag…
Language IdentificationComputational Approaches to Arabic-English Code-Switching
Natural Language Processing (NLP) is a vital computational method for addressing language processing, analysis, and generation. NLP tasks form the core of many daily applications, from automatic text correction to speech…
Data AugmentationLanguage Identificationnamed-entity-recognitionNamed Entity Recognition+4Generation through the lens of learning theory
We study generation through the lens of statistical learning theory. First, we abstract and formalize the results of Gold [1967], Angluin [1979], Angluin [1980] and Kleinberg and Mullainathan [2024] in terms of a binary …
Language IdentificationLearning TheoryA Multi-Task Text Classification Pipeline with Natural Language Explanations: A User-Centric Evaluation in Sentiment Analysis and Offensive Language Identification in Greek Tweets
Interpretability is a topic that has been in the spotlight for the past few years. Most existing interpretability techniques produce interpretations in the form of rules or feature importance. These interpretations, whil…
Feature ImportanceLanguage IdentificationLanguage ModelingLanguage Modelling+4From N-grams to Pre-trained Multilingual Models For Language Identification
In this paper, we investigate the use of N-gram models and Large Pre-trained Multilingual models for Language Identification (LID) across 11 South African languages. For N-gram models, this study shows that effective dat…
Language IdentificationXLM-REfficiently Identifying Low-Quality Language Subsets in Multilingual Datasets: A Case Study on a Large-Scale Multilingual Audio Dataset
Curating datasets that span multiple languages is challenging. To make the collection more scalable, researchers often incorporate one or more imperfect classifiers in the process, like language identification models. Th…
Language IdentificationAfriHuBERT: A self-supervised speech representation model for African languages
In this work, we present AfriHuBERT, an extension of mHuBERT-147, a compact self-supervised learning (SSL) model pretrained on 147 languages. While mHuBERT-147 covered 16 African languages, we expand this to 1,226 throug…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Cross-corpusLanguage Identification+4Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking
Multilingual Automatic Speech Recognition (ASR) models are typically evaluated in a setting where the ground-truth language of the speech utterance is known, however, this is often not the case for most practical setting…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Language IdentificationRe-Ranking+3Leveraging Open-Source Large Language Models for Native Language Identification
Native Language Identification (NLI) - the task of identifying the native language (L1) of a person based on their writing in the second language (L2) - has applications in forensics, marketing, and second language acqui…
Feature EngineeringLanguage AcquisitionLanguage IdentificationMarketing+2Enhancing Code-Switching Speech Recognition with LID-Based Collaborative Mixture of Experts Model
Due to the inherent difficulty in modeling phonetic similarities across different languages, code-switching speech recognition presents a formidable challenge. This study proposes a Collaborative-MoE, a Mixture of Expert…
Language IdentificationMixture-of-Expertsspeech-recognitionSpeech RecognitionLiterary and Colloquial Dialect Identification for Tamil using Acoustic Features
The evolution and diversity of a language is evident from it's various dialects. If the various dialects are not addressed in technological advancements like automatic speech recognition and speech synthesis, there is a …
Automatic Speech RecognitionDialect IdentificationLanguage Identificationspeech-recognition+2Language-Informed Beam Search Decoding for Multilingual Machine Translation
Beam search decoding is the de-facto method for decoding auto-regressive Neural Machine Translation (NMT) models, including multilingual NMT where the target language is specified as an input. However, decoding multiling…
Language IdentificationMachine TranslationNMTTranslationSpeech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond
We present Speech-MASSIVE, a multilingual Spoken Language Understanding (SLU) dataset comprising the speech counterpart for a portion of the MASSIVE textual corpus. Speech-MASSIVE covers 12 languages from different famil…
BenchmarkingLanguage Identificationslot-fillingSlot Filling+1Towards Generalized Offensive Language Identification
The prevalence of offensive content on the internet, encompassing hate speech and cyberbullying, is a pervasive issue worldwide. Consequently, it has garnered significant attention from the machine learning (ML) and natu…
Language IdentificationA Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models
Recent studies have highlighted the potential of exploiting parallel corpora to enhance multilingual large language models, improving performance in both bilingual tasks, e.g., machine translation, and general-purpose ta…
Language IdentificationMachine TranslationSentencetext-classification+2SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASR
In this work, we propose a Switch-Conformer-based MoE system named SC-MoE for unified streaming and non-streaming code-switching (CS) automatic speech recognition (ASR), where we design a streaming MoE layer consisting o…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Computational EfficiencyDecoder+4Script-Agnostic Language Identification
Language identification is used as the first step in many data collection and crawling efforts because it allows us to sort online text into language-specific buckets. However, many modern languages, such as Konkani, Kas…
Language IdentificationRapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Prompting
End-to-end multilingual speech recognition models handle multiple languages through a single model, often incorporating language identification to automatically detect the language of incoming speech. Since the common sc…
DecoderLanguage Identificationspeech-recognitionSpeech Recognition