Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review
Recent technological advancements have enhanced our ability to collect and analyze rich multimodal data (e.g., speech, video, and eye gaze) to better inform learning and training experiences. While previous reviews have focused on parts of the multimodal pipeline (e.g., conceptual models and data fusion), a comprehensive literature review on the methods informing multimodal learning and training environments has not been conducted. This literature review provides an in-depth analysis of research methods in these environments, proposing a taxonomy and framework that encapsulates recent methodological advances in this field and characterizes the multimodal domain in terms of five modality groups: Natural Language, Video, Sensors, Human-Centered, and Environment Logs. We introduce a novel data fusion category -- mid fusion -- and a graph-based technique for refining literature reviews, termed citation graph pruning. Our analysis reveals that leveraging multiple modalities offers a more holistic understanding of the behaviors and outcomes of learners and trainees. Even when multimodality does not enhance predictive accuracy, it often uncovers patterns that contextualize and elucidate unimodal data, revealing subtleties that a single modality may miss. However, there remains a need for further research to bridge the divide between multimodal learning and training studies and foundational AI research.
Code (0)
등록된 구현이 없습니다.
Tasks
Systematic Literature ReviewSimilar Papers 제목 키워드 기반
LLMGeo: Benchmarking Large Language Models on Image Geolocation In-the-wild
Image geolocation is a critical task in various image-understanding applications. However, existing methods often fail when analyzing challenging, in-the-wild images. Inspired by the exceptional background knowledge of m…
BenchmarkingA First Step in Using Machine Learning Methods to Enhance Interaction Analysis for Embodied Learning Environments
Investigating children's embodied learning in mixed-reality environments, where they collaboratively simulate scientific processes, requires analyzing complex multimodal data to interpret their learning and coordination …
Mixed RealityAdaptation through prediction: multisensory active inference torque control
Adaptation to external and internal changes is major for robotic systems in uncertain environments. Here we present a novel multisensory active inference torque controller for industrial arms that shows how prediction ca…
PredictionOnline Continual Learning: A Systematic Literature Review of Approaches, Challenges, and Benchmarks
Online Continual Learning (OCL) is a critical area in machine learning, focusing on enabling models to adapt to evolving data streams in real-time while addressing challenges such as catastrophic forgetting and the stabi…
Continual Learningimage-classificationImage Classificationobject-detection+3UniFinEval: Towards Unified Evaluation of Financial Multimodal Models across Text, Images and Videos
Multimodal large language models are playing an increasingly significant role in empowering the financial domain, however, the challenges they face, such as multimodal and high-density information and cross-modal multi-h…