An Evolutionary Network Architecture Search Framework with Adaptive Multimodal Fusion for Hand Gesture Recognition
Hand gesture recognition (HGR) based on multimodal data has attracted considerable attention owing to its great potential in applications. Various manually designed multimodal deep networks have performed well in multimodal HGR (MHGR), but most of existing algorithms require a lot of expert experience and time-consuming manual trials. To address these issues, we propose an evolutionary network architecture search framework with the adaptive multimodel fusion (AMF-ENAS). Specifically, we design an encoding space that simultaneously considers fusion positions and ratios of the multimodal data, allowing for the automatic construction of multimodal networks with different architectures through decoding. Additionally, we consider three input streams corresponding to intra-modal surface electromyography (sEMG), intra-modal accelerometer (ACC), and inter-modal sEMG-ACC. To automatically adapt to various datasets, the ENAS framework is designed to automatically search a MHGR network with appropriate fusion positions and ratios. To the best of our knowledge, this is the first time that ENAS has been utilized in MHGR to tackle issues related to the fusion position and ratio of multimodal data. Experimental results demonstrate that AMF-ENAS achieves state-of-the-art performance on the Ninapro DB2, DB3, and DB7 datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Gesture RecognitionHand Gesture RecognitionHand-Gesture RecognitionSimilar Papers 제목 키워드 기반
A Modality-Aware Cooperative Co-Evolutionary Framework for Multimodal Graph Neural Architecture Search
Co-exploitation attacks on software vulnerabilities pose severe risks to enterprises, a threat that can be mitigated by analyzing heterogeneous and multimodal vulnerability data. Multimodal graph neural networks (MGNNs) …
Neural Architecture SearchHierarchical Adaptive Expert for Multimodal Sentiment Analysis
Multimodal sentiment analysis has emerged as a critical tool for understanding human emotions across diverse communication channels. While existing methods have made significant strides, they often struggle to effectivel…
Emotion RecognitionEvolutionary AlgorithmsMultimodal Sentiment AnalysisMulti-Task Learning+2SONATA: Self-adaptive Evolutionary Framework for Hardware-aware Neural Architecture Search
Recent advancements in Artificial Intelligence (AI), driven by Neural Networks (NN), demand innovative neural architecture designs, particularly within the constrained environments of Internet of Things (IoT) systems, to…
Evolutionary AlgorithmsHardware Aware Neural Architecture SearchNeural Architecture SearchAn Adaptive Balance Search Based Complementary Heterogeneous Particle Swarm Optimization Architecture
A series of modified cognitive-only particle swarm optimization (PSO) algorithms effectively mitigate premature convergence by constructing distinct vectors for different particles. However, the underutilization of these…
Co-Evolutionary Multi-Modal Alignment via Structured Adversarial Evolution
Adversarial behavior plays a central role in aligning large language models with human values. However, existing alignment methods largely rely on static adversarial settings, which fundamentally limit robustness, partic…