Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioning
Text-to-image diffusion models have achieved remarkable progress in image synthesis, yet can exhibit memorization by closely reproducing individual training examples. Effective mitigation must preserve useful prompt information to guide alternative depictions. We introduce a training-free method that redistributes cross-attention with Gaussian smoothing before reinforcing content-token contributions and attenuating padding contributions, without additional denoiser evaluations. With this intervention, stronger content conditioning can improve prompt alignment at comparable training-image similarity. A local analysis identifies when reinforcement preserves shared value information while redistribution reduces localized attention mass. On Stable Diffusion v1.4 and v2.0, all evaluated smoothing widths lie on the empirical Pareto frontiers for training-image similarity versus both prompt alignment and image preference. A configuration selected on Stable Diffusion reduces template reproduction in DeepFloyd IF without further tuning. These findings support jointly controlling conditioning allocation and strength to generate prompt-consistent alternatives.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models
Diffusion models have achieved remarkable success in Text-to-Image generation tasks, leading to the development of many commercial models. However, recent studies have reported that diffusion models often generate replic…
Image GenerationMemorizationText to Image GenerationText-to-Image GenerationCapacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models
In this work, we present compelling evidence that controlling model capacity during fine-tuning can effectively mitigate memorization in diffusion models. Specifically, we demonstrate that adopting Parameter-Efficient Fi…
Memorizationparameter-efficient fine-tuningSteering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion
Text-to-image diffusion models often memorize training data, revealing a fundamental failure to generalize beyond the training set. Current mitigation strategies typically sacrifice image quality or prompt alignment to r…
Reinforcement LearningDemystifying Foreground-Background Memorization in Diffusion Models
Diffusion models (DMs) memorize training images and can reproduce near-duplicates during generation. Current detection methods identify verbatim memorization but fail to capture two critical aspects: quantifying partial …
Finding DoRI: Discovery of Retained Images in Diffusion Models
Text-to-image diffusion models (DMs) have achieved remarkable success in image generation. However, concerns about data privacy and intellectual property remain due to their potential to inadvertently memorize and replic…
Image Generation