A Graph-based Approach for Contextual Text Normalization

Social Text Normalization using Contextual Graph Random Walks

On the Expressive Power of Contextual Relations in Transformers

Transformer architectures have achieved remarkable empirical success in modeling contextual relations, yet a clear understanding of their expressive power is still lacking. In this work, we introduce a measure-theoretic …

Semi-supervised Contextual Historical Text Normalization

Historical text normalization, the task of mapping historical word forms to their modern counterparts, has recently attracted a lot of interest (Bollmann, 2019; Tang et al., 2018; Lusetti et al., 2018; Bollmann et al., 2…

Effect of Post-processing on Contextualized Word Representations

Post-processing of static embedding has beenshown to improve their performance on both lexical and sequence-level tasks. However, post-processing for contextualized embeddings is an under-studied problem. In this work, w…

On Separate Normalization in Self-supervised Transformers

Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normaliz…