CounterFlow: A Two-Phase Inference-Time Sampling for Counterfactual Video Foley Generation
We investigate Counterfactual Video Foley Generation, which aims to adopt a sound-source identity that contradicts the visual evidence while remaining temporally synchronized to a silent video. Existing Video&Text-to-Audio (VT2A) models struggle with this, often remaining anchored to the visually implied sound source when video and text contents disagree. We present ConterFlow, an inference-time dual-phase sampling scheme for pretrained flow-matching VT2A models. Phase 1 builds a video-derived temporal structure while suppressing the visually implied source; Phase 2 drops video conditioning to focus entirely on shaping audio timbre toward the target prompt. ConterFlow substantially improves counterfactual Video Foley generation compared to naive negative prompting and state-of-the-art baselines. To evaluate replacement quality, we propose a metric leveraging a text-audio co-embedding space to measure both target-prompt evidence and residual visually implied source leakage. Video demonstrations and code are available at https://gyubin-lee.github.io/counterflow-demo/
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
CounterFlowNet: From Minimal Changes to Meaningful Counterfactual Explanations
Counterfactual explanations (CFs) provide human-interpretable insights into model's predictions by identifying minimal changes to input features that would alter the model's output. However, existing methods struggle to …
Empowering Language Understanding with Counterfactual Reasoning
Present language understanding methods have demonstrated extraordinary ability of recognizing patterns in texts via machine learning. However, existing methods indiscriminately use the recognized patterns in the testing …
counterfactualCounterfactual ReasoningNatural Language InferenceSentiment AnalysisCounterfactual Inference under Thompson Sampling
Recommender systems exemplify sequential decision-making under uncertainty, strategically deciding what content to serve to users, to optimise a range of potential objectives. To balance the explore-exploit trade-off suc…
Causal InferencecounterfactualCounterfactual InferenceDecision Making+4Imputation of Counterfactual Outcomes when the Errors are Predictable
A crucial input into causal inference is the imputed counterfactual outcome. Imputation error can arise because of sampling uncertainty from estimating the prediction model using the untreated observations, or from out-o…
Causal InferencecounterfactualImputationTime SeriesTowards Multimodal Sentiment Analysis Debiasing via Bias Purification
Multimodal Sentiment Analysis (MSA) aims to understand human intentions by integrating emotion-related clues from diverse modalities, such as visual, language, and audio. Unfortunately, the current MSA task invariably su…
counterfactualCounterfactual InferenceMultimodal Sentiment AnalysisSentiment Analysis