PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding
Large language models (LM) based on Transformers allow to generate plausible long texts. In this paper, we explore how this generation can be further controlled at decoding time to satisfy certain constraints (e.g. being non-toxic, conveying certain emotions, using a specific writing style, etc.) without fine-tuning the LM. Precisely, we formalize constrained generation as a tree exploration process guided by a discriminator that indicates how well the associated sequence respects the constraint. This approach, in addition to being easier and cheaper to train than fine-tuning the LM, allows to apply the constraint more finely and dynamically. We propose several original methods to search this generation tree, notably the Monte Carlo Tree Search (MCTS) which provides theoretical guarantees on the search efficiency, but also simpler methods based on re-ranking a pool of diverse sequences using the discriminator scores. These methods are evaluated, with automatic and human-based metrics, on two types of constraints and languages: review polarity and emotion control in French and English. We show that discriminator-guided MCTS decoding achieves state-of-the-art results without having to tune the language model, in both tasks and languages. We also demonstrate that other proposed decoding methods based on re-ranking can be really effective when diversity among the generated propositions is encouraged.
Code (1)
Tasks
Language ModellingRe-RankingSimilar Papers 제목 키워드 기반
PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding
Large language models (LM) based on Transformers allow to generate plausible long texts. In this paper, we explore how this generation can be further controlled at decoding time to satisfy certain constraints (eg. being …
Language ModellingRe-RankingImproving Conditional Sequence Generative Adversarial Networks by Stepwise Evaluation
Sequence generative adversarial networks (SeqGAN) have been used to improve conditional sequence generation tasks, for example, chit-chat dialogue generation. To stabilize the training of SeqGAN, Monte Carlo tree search …
Dialogue GenerationTidiness Score-Guided Monte Carlo Tree Search for Visual Tabletop Rearrangement
In this paper, we present the tidiness score-guided Monte Carlo tree search (TSMCTS), a novel framework designed to address the tabletop tidying up problem using only an RGB-D camera. We address two major problems for ta…
Décodage guidé par un discriminateur avec le Monte Carlo Tree Search pour la génération de texte contrainte (Discriminator-guided decoding with Monte Carlo Tree Search for constrained text generation )
Dans cet article, nous explorons comment contrôler la génération de texte au moment du décodage pour satisfaire certaines contraintes (e.g. être non toxique, transmettre certaines émotions...), sans nécessiter de ré-entr…
Text GenerationTrust-Region Twisted Policy Improvement
Monte-Carlo tree search (MCTS) has driven many recent breakthroughs in deep reinforcement learning (RL). However, scaling MCTS to parallel compute has proven challenging in practice which has motivated alternative planne…
Deep Reinforcement LearningReinforcement Learning (RL)