Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain
Large language models (LLMs) make it plausible to build systems that improve through self-evolving loops, but many existing proposals are better understood as self-play and often plateau quickly. A central failure mode is that the loop synthesises more data without increasing learnable information for the next iteration. Through experiments on a self-play coding task, we reveal that sustainable self-evolution requires a self-synthesised data pipeline with learnable information that increases across iterations. We identify triadic roles that self-evolving LLMs play: the Proposer, which generates tasks; the Solver, which attempts solutions; and the Verifier, which provides training signals, and we identify three system designs that jointly target learnable information gain from this triadic roles perspective. Asymmetric co-evolution closes a weak-to-strong-to-weak loop across roles. Capacity growth expands parameter and inference-time budgets to match rising learnable information. Proactive information seeking introduces external context and new task sources that prevent saturation. Together, these modules provide a measurable, system-level path from brittle self-play dynamics to sustained self-evolution.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games
Recent work has established that regularized policy gradient methods such as PPO, when used in self-play, can match or exceed specialized game-theoretic algorithms for solving two-player zero-sum imperfect-information ga…
SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks
Self-play can train language models without external supervision. However, existing methods require rule-checkable answers, leaving open-ended tasks dependent on curated prompts or frontier-model judges. We introduce SCO…
SelfReplay: Adapting Self-Supervised Sensory Models via Adaptive Meta-Task Replay
Self-supervised learning has emerged as a method for utilizing massive unlabeled data for pre-training models, providing an effective feature extractor for various mobile sensing applications. However, when deployed to e…
Computational EfficiencyDomain AdaptationMeta-LearningSelf-Supervised LearningSelf-Generative Adversarial Fine-Tuning for Large Language Models
Fine-tuning large language models (LLMs) for alignment typically relies on supervised fine-tuning or reinforcement learning from human feedback, both limited by the cost and scarcity of high-quality annotations. Recent s…
Reinforcement LearningTool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Data
Large language models (LLMs) are becoming the foundation for autonomous agents that can use tools to solve complex tasks. Reinforcement learning (RL) has emerged as a common approach for injecting such agentic capabiliti…
Reinforcement Learning