P-Masking: Power Law Masking Improves Multi-attribute Controlled Generation
We introduce LingGen, a novel approach for controlled text generation that offers precise control over a wide array of linguistic attributes, even as the number of attributes varies. LingGen employs a dynamic P-MASKING strategy, which samples masking rates from a power law distribution during training. This innovative approach enables the model to develop robust representations and adapt its attribute control capabilities across a variable number of attributes, from a single attribute to multiple complex configurations. The P-MASKING technique enhances LingGen's ability to manage different levels of attribute visibility, resulting in superior performance in multi-attribute generation tasks. Our experiments demonstrate that LingGen surpasses current state-of-the-art models in both attribute control accuracy and text fluency, particularly excelling in scenarios with varying attribute demands. Additionally, our ablation studies highlight the effectiveness of P-MASKING and the influence of different base language models on performance. These findings demonstrate LingGen's potential for applications requiring precise and adaptable control over multiple linguistic attributes in text generation.
Code (0)
등록된 구현이 없습니다.
Tasks
AttributeText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Extreme Multi-Label Classification with Label Masking for Product Attribute Value Extraction
Although most studies have treated attribute value extraction (AVE) as named entity recognition, these approaches are not practical in real-world e-commerce platforms because they perform poorly, and require canonicaliza…
AttributeAttribute Value ExtractionExtreme Multi-Label ClassificationMulti-Label Classification+4Token Masking Improves Transformer-Based Text Classification
While transformer-based models achieve strong performance on text classification, we explore whether masking input tokens can further enhance their effectiveness. We propose token masking regularization, a simple yet the…
AttributeClassificationLanguage IdentificationSentiment Analysis+2PMI-Masking: Principled masking of correlated spans
Masking tokens uniformly at random constitutes a common flaw in the pretraining of Masked Language Models (MLMs) such as BERT. We show that such uniform masking allows an MLM to minimize its training objective by latchin…
Structured-Noise Masked Modeling for Video, Audio and Beyond
Masked modeling has emerged as a powerful self-supervised learning framework, but existing methods largely rely on random masking, disregarding the structural properties of different modalities. In this work, we introduc…
Representation LearningSelf-Supervised LearningUnsupervised Text Style Transfer via LLMs and Attention Masking with Multi-way Interactions
Unsupervised Text Style Transfer (UTST) has emerged as a critical task within the domain of Natural Language Processing (NLP), aiming to transfer one stylistic aspect of a sentence into another style without changing its…
In-Context LearningKnowledge DistillationSentenceStyle Transfer+2