NormEval: A Unified Multi-Metric Framework for Evaluating Semantic Fidelity in Text Normalization
Text normalization methods such as stemming and lemmatization are fundamental components of NLP pipelines. As new normalization tools are developed for diverse languages, evaluation methodologies remain fragmented, relying on Compression Ratio, downstream accuracy, or sequence-to-sequence prediction scores in isolation, failing to distinguish between beneficial vocabulary reduction and harmful semantic distortion. Moreover, text normalization underpins intelligent systems in high-stakes domains, including clinical decision support and legal document analysis, and principled evaluation methodology is essential. This paper proposes NormEval, a unified, multilingual evaluation framework comprising five complementary metrics: Compression Ratio (CR), Model Performance Delta (MPD), Information Retention Score (IRS), Algorithm Effectiveness Score (AES), and Average Normalized Levenshtein Distance (ANLD). These metrics assess normalization quality across three dimensions: macro-level efficiency, downstream utility, and micro-level morphological fidelity. The framework operationalizes a Safety Gate hypothesis: ANLD functions as an intrinsic structural hygiene check, utilizing character-level divergence ($Δ$) to reveal aggressive mutations that macro-level embeddings and downstream tasks mask. Comprehensive ablation experiments on both Bangla and English datasets show that all the components are indispensable, and that the removal of any individual metric leads to a decrease in at least one evaluation aspect, which ultimately results in misleading algorithm rankings.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
KG-EDAS: A Meta-Metric Framework for Evaluating Knowledge Graph Completion Models
Knowledge Graphs (KGs) enable applications in various domains such as semantic search, recommendation systems, and natural language processing. KGs are often incomplete, missing entities and relations, an issue addressed…
Knowledge Graph CompletionRecommendation SystemsKnowledge GraphsProMoral-Bench: Evaluating Prompting Strategies for Moral Reasoning and Safety in LLMs
Prompt design significantly impacts the moral competence and safety alignment of large language models (LLMs), yet empirical comparisons remain fragmented across datasets and models.We introduce ProMoral-Bench, a unified…
Prompt EngineeringRevisiting SLO and Goodput Metrics in LLM Serving
Large language models (LLMs) have achieved remarkable performance and are widely deployed in various applications, while the serving of LLM inference has raised concerns about user experience and serving throughput. Acco…
A Unified Framework of Surrogate Loss by Refactoring and Interpolation
We introduce UniLoss, a unified framework to generate surrogate losses for training deep networks with gradient descent, reducing the amount of manual design of task-specific surrogate losses. Our key observation is that…
Evaluating Disentanglement of Structured Latent Representations
We introduce the first metric for evaluating disentanglement at individual hierarchy levels of a structured latent representation. Applied to object-centric generative models, this offers a systematic, unified approach t…
DisentanglementObject