Deep dive into CoCon - A Self Supervised approach for Controlled Text Generation
Transformer-based language models ([1] Vaswani et.al, 2017) have stirred transfer-based learning in NLP and have improved the performance of several NLP tasks. The preliminary step involves pretraining a language model on a large amount of text on the web. Research on steering a pretrained language model to enable fine-grained control over the content and sentiment of output is still under active exploration and has great potential in various applications such as story generation, search engines, etc. This blog post discusses a paper which proposes a content conditioner when trained auto-regressively alongside a Large pretrained language model (like GPT-2) provides the capability to control text at a fine-grained level.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingStory GenerationText GenerationSimilar Papers 제목 키워드 기반
CoCon: A Self-Supervised Approach for Controlled Text Generation
Pretrained Transformer-based language models (LMs) display remarkable natural language generation capabilities. With their immense potential, controlling text generation of such LMs is getting attention. While there are …
Text GenerationNoisyCoconut: Counterfactual Consensus via Latent Space Reasoning
This paper presents NoisyCoconut, a novel inference-time method that enhances large language model (LLM) reliability by manipulating internal representations. Unlike fine-tuning methods that require extensive retraining,…
Mathematical ReasoningModeling Multiple Views via Implicitly Preserving Global Consistency and Local Complementarity
While self-supervised learning techniques are often used to mining implicit knowledge from unlabeled data via modeling multiple views, it is unclear how to perform effective representation learning in a complex and incon…
Representation LearningSelf-Supervised LearningDetection of Adulteration in Coconut Milk using Infrared Spectroscopy and Machine Learning
In this paper, we propose a system for detecting adulteration in coconut milk, utilizing infrared spectroscopy. The machine learning-based proposed system comprises three phases: preprocessing, feature extraction, and cl…
Why Limit the Residual Stream to Layers and Not Tokens? Persistent Memory for Continuous Latent Reasoning
Large language models (LLMs) have demonstrated remarkable reasoning abilities on mathematical and multi-hop planning tasks. The CoCoNuT (Chain of Continuous Thought) paradigm~\cite{hao2024coconut} extends this by enablin…