Towards Shared Datasets for Normalization Research
In this paper we present a Dutch and English dataset that can serve as a gold standard for evaluating text normalization approaches. With the combination of text messages, message board posts and tweets, these datasets represent a variety of user generated content. All data was manually normalized to their standard form using newly-developed guidelines. We perform automatic lexical normalization experiments on these datasets using statistical machine translation techniques. We focus on both the word and character level and find that we can improve the BLEU score with ca. 20{\%} for both languages. In order for this user generated content data to be released publicly to the research community some issues first need to be resolved. These are discussed in closer detail by focussing on the current legislation and by investigating previous similar data collection projects. With this discussion we hope to shed some light on various difficulties researchers are facing when trying to share social media data.
Code (0)
등록된 구현이 없습니다.
Tasks
Domain AdaptationLexical NormalizationMachine TranslationOpinion MiningText NormalizationTranslationSimilar Papers 제목 키워드 기반
TweetNorm\_es: an annotated corpus for Spanish microtext normalization
In this paper we introduce TweetNorm{\_}es, an annotated corpus of tweets in Spanish language, which we make publicly available under the terms of the CC-BY license. This corpus is intended for development and testing of…
Domain AdaptationLexical NormalizationMachine TranslationSentiment AnalysisShared Differential Clustering across Single-cell RNA Sequencing Datasets with the Hierarchical Dirichlet Process
Single-cell RNA sequencing (scRNA-seq) is powerful technology that allows researchers to understand gene expression patterns at the single-cell level. However, analysing scRNA-seq data is challenging due to issues and bi…
ClusteringImputationStar+: A New Multi-Domain Model for CTR Prediction
In this paper, we introduce Star+, a novel multi-domain model for click-through rate (CTR) prediction inspired by the Star model. Traditional single-domain approaches and existing multi-task learning techniques face chal…
Click-Through Rate PredictionMulti-Task LearningPredictionRecommendation SystemsÚFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5
We present the winning entry to the Multilingual Lexical Normalization (MultiLexNorm) shared task at W-NUT 2021 (van der Goot et al., 2021a), which evaluates lexical-normalization systems on 12 social media datasets in 1…
Dependency ParsingLanguage ModelingLanguage ModellingLexical NormalizationDS4DH at #SMM4H 2023: Zero-Shot Adverse Drug Events Normalization using Sentence Transformers and Reciprocal-Rank Fusion
This paper outlines the performance evaluation of a system for adverse drug event normalization, developed by the Data Science for Digital Health (DS4DH) group for the Social Media Mining for Health Applications (SMM4H) …
Sentence