Counterfactual Off-Policy Training for Neural Response Generation
Open-domain dialogue generation suffers from the data insufficiency problem due to the vast size of potential responses. In this paper, we propose to explore potential responses by counterfactual reasoning. Given an observed response, the counterfactual reasoning model automatically infers the outcome of an alternative policy that could have been taken. The resulting counterfactual response synthesized in hindsight is of higher quality than the response synthesized from scratch. Training on the counterfactual responses under the adversarial learning framework helps to explore the high-reward area of the potential response space. An empirical study on the DailyDialog dataset shows that our approach significantly outperforms the HRED model as well as the conventional adversarial learning approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualCounterfactual ReasoningDialogue GenerationResponse GenerationSimilar Papers 제목 키워드 기반
Counterfactual Off-Policy Training for Neural Dialogue Generation
Open-domain dialogue generation suffers from the data insufficiency problem due to the vast size of potential responses. In this paper, we propose to explore potential responses by counterfactual reasoning. Given an obse…
counterfactualCounterfactual ReasoningDialogue GenerationCROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation
On-policy distillation (OPD) supervises a student language model on trajectories sampled from its current policy, but assigns equal credit to response tokens with unequal supervision value. Selective OPD addresses this l…
CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
Rubric-based reinforcement learning enriches language model training by evaluating model outputs against explicit criteria. Yet in GRPO-style pipelines, these structured judgments are reduced to a scalar response-level r…
Reinforcement LearningCounterfactual Inference in Duration Models with Random Censoring
We propose a counterfactual Kaplan-Meier estimator that incorporates exogenous covariates and unobserved heterogeneity of unrestricted dimensionality in duration models with random censoring. Under some regularity condit…
counterfactualCounterfactual InferenceCoFi-PGMA: Counterfactual Policy Gradients under Filtered Feedback for Multi-Agent LLMs
Large language model (LLM) deployments increasingly rely on multi-agent architectures in which multiple models either compete through routing mechanisms or collaborate to produce a final answer. In both settings, the lea…