Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesis

2025-04-14 · Zihao Liu, Mingwen Ou, Zunnan Xu, Jiaqi Huang, Haonan Han, Ronghui Li, Xiu Li

Automating the synthesis of coordinated bimanual piano performances poses significant challenges, particularly in capturing the intricate choreography between the hands while preserving their distinct kinematic signatures. In this paper, we propose a dual-stream neural framework designed to generate synchronized hand gestures for piano playing from audio input, addressing the critical challenge of modeling both hand independence and coordination. Our framework introduces two key innovations: (i) a decoupled diffusion-based generation framework that independently models each hand's motion via dual-noise initialization, sampling distinct latent noise for each while leveraging a shared positional condition, and (ii) a Hand-Coordinated Asymmetric Attention (HCAA) mechanism suppresses symmetric (common-mode) noise to highlight asymmetric hand-specific features, while adaptively enhancing inter-hand coordination during denoising. The system operates hierarchically: it first predicts 3D hand positions from audio features and then generates joint angles through position-aware diffusion models, where parallel denoising streams interact via HCAA. Comprehensive evaluations demonstrate that our framework outperforms existing state-of-the-art methods across multiple metrics.

📄 PDF Abstract BibTeX arXiv:2504.09885

Code (0)

등록된 구현이 없습니다.

Tasks

Denoising Motion Synthesis

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Safety-aware Semi-end-to-end Coordinated Decision Model for Voltage Regulation in Active Distribution Network

2023-05-24 · Linwei Sang, Yinliang Xu, Huan Long, Wenchuan Wu

Prediction plays a vital role in the active distribution network voltage regulation under the high penetration of photovoltaics. Current prediction models aim at minimizing individual prediction errors but overlook their…

Decision Making Prediction

Collaboration of AI Agents via Cooperative Multi-Agent Deep Reinforcement Learning

2019-06-30 · Niranjan Balachandar, Justin Dieter, Govardana Sachithanandam Ramachandran

There are many AI tasks involving multiple interacting agents where agents should learn to cooperate and collaborate to effectively perform the task. Here we develop and evaluate various multi-agent protocols to train ag…

counterfactual Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

CoorDex: Coordinating Body and Hand Priors for Continuous Dexterous Humanoid Loco-Manipulation

2026-06-22 · Sikai Li, Shuning Li, Zhenyu Wei, Yunchao Yao 외 arxiv

Humanoid loco-manipulation is often simplified into a stop-and-go process: walking to an object, stopping to manipulate it, and then resuming locomotion. It also commonly relies on low degree-of-freedom (DoF) end effecto…

Reinforcement Learning

Dual-Stream Diffusion Net for Text-to-Video Generation

2023-08-16 · Binhui Liu, Xin Liu, Anbo Dai, Zhiyong Zeng 외

With the emerging diffusion models, recently, text-to-video generation has aroused increasing attention. But an important bottleneck therein is that generative videos often tend to carry some flickers and artifacts. In t…

Text-to-Video Generation Video Generation

ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inference

2026-06-18 · Yang Tan, Junlong Tong, Linan Yue, Hao Wu 외 arxiv

Streaming VideoLLMs must continuously process incoming video while maintaining low query latency, making both video-ingestion throughput and query-time responsiveness critical for real-time deployment. Existing methods l…