Replace in Translation: Boost Concept Alignment in Counterfactual Text-to-Image
Text-to-Image (T2I) has been prevalent in recent years, with most common condition tasks having been optimized nicely. Besides, counterfactual Text-to-Image is obstructing us from a more versatile AIGC experience. For those scenes that are impossible to happen in real world and anti-physics, we should spare no efforts in increasing the factual feel, which means synthesizing images that people think very likely to be happening, and concept alignment, which means all the required objects should be in the same frame. In this paper, we focus on concept alignment. As controllable T2I models have achieved satisfactory performance for real applications, we utilize this technology to replace the objects in a synthesized image in latent space step-by-step to change the image from a common scene to a counterfactual scene to meet the prompt. We propose a strategy to instruct this replacing process, which is called as Explicit Logical Narrative Prompt (ELNP), by using the newly SoTA language model DeepSeek to generate the instructions. Furthermore, to evaluate models' performance in counterfactual T2I, we design a metric to calculate how many required concepts in the prompt can be covered averagely in the synthesized images. The extensive experiments and qualitative comparisons demonstrate that our strategy can boost the concept alignment in counterfactual T2I.
Code (0)
등록된 구현이 없습니다.
Tasks
Concept AlignmentcounterfactualMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Counterfactual Data Augmentation for Neural Machine Translation
We propose a data augmentation method for neural machine translation. It works by interpreting language models and phrasal alignment causally. Specifically, it creates augmented parallel translation corpora by generating…
counterfactualData AugmentationLanguage ModelingLanguage Modelling+2Concept-Level Explainability for Auditing & Steering LLM Responses
As large language models (LLMs) become widely deployed, concerns about their safety and alignment grow. An approach to steer LLM behavior, such as mitigating biases or defending against jailbreaks, is to identify which p…
Prompt EngineeringSemantic SimilaritySemantic Textual SimilarityText GenerationEnd-to-End Neural Word Alignment Outperforms GIZA++
Word alignment was once a core unsupervised learning task in natural language processing because of its essential role in training statistical machine translation (MT) models. Although unnecessary for training neural MT …
Machine TranslationTranslationWord AlignmentTowards generating more interpretable counterfactuals via concept vectors: a preliminary study on chest X-rays
An essential step in deploying medical imaging models is ensuring alignment with clinical knowledge and interpretability. We focus on mapping clinical concepts into the latent space of generative models to identify Conce…
Clinical KnowledgeInterpreting and Controlling Model Behavior via Constitutions for Atomic Concept Edits
We introduce a black-box interpretability framework that learns a verifiable constitution: a natural language summary of how changes to a prompt affect a model's specific behavior, such as its alignment, correctness, or …
Text-to-Image GenerationMathematical Reasoning