Papers Language Identification
“Language Identification” 태그가 달린 논문 852편 · 필터 해제
Scaling HuBERT for African Languages: From Base to Large and XL
Despite recent progress in multilingual speech processing, African languages remain under-represented in both research and deployed systems, particularly when it comes to strong, open-weight encoders that transfer well u…
Self-Supervised LearningLanguage IdentificationSpeech RecognitionLanguage Generation and Identification From Partial Enumeration: Tight Density Bounds and Topological Characterizations
The success of large language models (LLMs) has motivated formal theories of language generation and learning. We study the framework of \emph{language generation in the limit}, where an adversary enumerates strings from…
Language IdentificationLanguage Generation: Complexity Barriers and Implications for Learning
Kleinberg and Mullainathan showed that language generation in the limit is always possible at the level of computability: given enough positive examples, a learner can eventually generate data indistinguishable from a ta…
Language IdentificationA Characterization of List Language Identification in the Limit
We study the problem of language identification in the limit, where given a sequence of examples from a target language, the goal of the learner is to output a sequence of guesses for the target language such that all th…
Language IdentificationHPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models
We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely the largest generally available multilingu…
Language IdentificationMachine TranslationSFMS-ALR: Script-First Multilingual Speech Synthesis with Adaptive Locale Resolution
Intra-sentence multilingual speech synthesis (code-switching TTS) remains a major challenge due to abrupt language shifts, varied scripts, and mismatched prosody between languages. Conventional TTS systems are typically …
Language IdentificationSpeech SynthesisA Multi-lingual Dataset of Classified Paragraphs from Open Access Scientific Publications
We present a dataset of 833k paragraphs extracted from CC-BY licensed scientific publications, classified into four categories: acknowledgments, data mentions, software/code mentions, and clinical trial mentions. The par…
Language IdentificationText ClassificationDIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching
Language Identification (LID) is a core task in multilingual NLP, yet current systems often overfit to clean, monolingual data. This work introduces DIVERS-BENCH, a comprehensive evaluation of state-of-the-art LID models…
Language IdentificationLaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQA
As large vision language models (VLMs) advance, their capabilities in multilingual visual question answering (mVQA) have significantly improved. Chain-of-thought (CoT) reasoning has been proven to enhance interpretabilit…
Visual Question AnsweringLanguage IdentificationMultimodal ReasoningLogical ReasoningBilingual Word Level Language Identification for Omotic Languages
Language identification is the task of determining the languages for a given text. In many real world scenarios, text may contain more than one language, particularly in multilingual communities. Bilingual Language Ident…
Language IdentificationTokenVerse++: Towards Flexible Multitask Learning with Dynamic Task Activation
Token-based multitasking frameworks like TokenVerse require all training utterances to have labels for all tasks, hindering their ability to leverage partially annotated datasets and scale effectively. We propose TokenVe…
Language IdentificationOvercoming Low-Resource Barriers in Tulu: Neural Models and Corpus Creation for OffensiveLanguage Identification
Tulu, a low-resource Dravidian language predominantly spoken in southern India, has limited computational resources despite its growing digital presence. This study presents the first benchmark dataset for Offensive Lang…
Language IdentificationLeveraging Zipformer Model for Effective Language Identification in Code-Switched Child-Directed Speech
Code-switching and language identification in child-directed scenarios present significant challenges, particularly in bilingual environments. This paper addresses this challenge by using Zipformer to handle the nuances …
Language IdentificationLanguage Detection by Means of the Minkowski Norm: Identification Through Character Bigrams and Frequency Analysis
The debate surrounding language identification has gained renewed attention in recent years, especially with the rapid evolution of AI-powered language models. However, the non-AI-based approaches to language identificat…
Language IdentificationILID: Native Script Language Identification for Indian Languages
The language identification task is a crucial fundamental step in NLP. Often it serves as a pre-processing step for widely used NLP applications such as multilingual machine translation, information retrieval, question a…
Language IdentificationInformation RetrievalMachine TranslationText SummarizationSparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages
Understanding the multilingual mechanisms of large language models (LLMs) provides insight into how they process different languages, yet this remains challenging. Existing studies often focus on individual neurons, but …
Language IdentificationOn Barriers to Archival Audio Processing
In this study, we leverage a unique UNESCO collection of mid-20th century radio recordings to probe the robustness of modern off-the-shelf language identification (LID) and speaker recognition (SR) methods, especially wi…
Language IdentificationSpeaker RecognitionKAConvText: Novel Approach to Burmese Sentence Classification using Kolmogorov-Arnold Convolution
This paper presents the first application of Kolmogorov-Arnold Convolution for Text (KAConvText) in sentence classification, addressing three tasks: imbalanced binary hate speech detection, balanced multiclass news class…
Sentence ClassificationLanguage IdentificationHate Speech DetectionNews ClassificationmSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks
Large Language models (LLMs) have demonstrated impressive performance on a wide range of tasks, including in multimodal settings such as speech. However, their evaluation is often limited to English and a few high-resour…
Language IdentificationQuestion Answeringtext-classificationText ClassificationNeighbors and relatives: How do speech embeddings reflect linguistic connections across the world?
Investigating linguistic relationships on a global scale requires analyzing diverse features such as syntax, phonology and prosody, which evolve at varying rates influenced by internal diversification, language contact, …
Language Identification