Designing, Playing, and Performing with a Vision-based Mouth Interface
The role of the face and mouth in speech production as well asnon-verbal communication suggests the use of facial action tocontrol musical sound. Here we document work on theMouthesizer, a system which uses a headworn miniaturecamera and computer vision algorithm to extract shapeparameters from the mouth opening and output these as MIDIcontrol changes. We report our experience with variousgesture-to-sound mappings and musical applications, anddescribe a live performance which used the Mouthesizerinterface.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playing
Text-based role-playing models can imitate character styles, but often fail to capture scene atmosphere and evolving tension, which are crucial for immersive applications such as VR games and interactive narratives. We s…
Reinforcement LearningResponse GenerationAI for Games in the Foundation Model Era
Foundation models, alongside advances in learned game-world models, are reshaping AI across the game lifecycle. Beyond playing games, recent systems model players and game dynamics, support design and development, adapt …
A Framework for Interactive Knowledge-Aided Machine Teaching
Machine Teaching (MT) is an interactive process where humans train a machine learning model by playing the role of a teacher. The process of designing an MT system involves decisions that can impact both efficiency of hu…
text-classificationText ClassificationDenoising instrumented mouthguard measurements of head impact kinematics with a convolutional neural network
Wearable sensors for measuring head kinematics can be noisy due to imperfect interfaces with the body. Mouthguards are used to measure head kinematics during impacts in traumatic brain injury (TBI) studies, but deviation…
DenoisingFirst Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization
How can we train an assistive human-machine interface (e.g., an electromyography-based limb prosthesis) to translate a user's raw command signals into the actions of a robot or computer when there is no prior mapping, we…