MORAL: A Multimodal Reinforcement Learning Framework for Decision Making in Autonomous Laboratories
We propose MORAL (a multimodal reinforcement learning framework for decision making in autonomous laboratories) that enhances sequential decision-making in autonomous robotic laboratories through the integration of visual and textual inputs. Using the BridgeData V2 dataset, we generate fine-tuned image captions with a pretrained BLIP-2 vision-language model and combine them with visual features through an early fusion strategy. The fused representations are processed using Deep Q-Network (DQN) and Proximal Policy Optimization (PPO) agents. Experimental results demonstrate that multimodal agents achieve a 20% improvement in task completion rates and significantly outperform visual-only and textual-only baselines after sufficient training. Compared to transformer-based and recurrent multimodal RL models, our approach achieves superior performance in cumulative reward and caption quality metrics (BLEU, METEOR, ROUGE-L). These results highlight the impact of semantically aligned language cues in enhancing agent learning efficiency and generalization. The proposed framework contributes to the advancement of multimodal reinforcement learning and embodied AI systems in dynamic, real-world environments.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingImage CaptioningLanguage ModelingLanguage Modellingreinforcement-learningReinforcement LearningSequential Decision MakingSimilar Papers 제목 키워드 기반
Addressing Moral Uncertainty using Large Language Models for Ethical Decision-Making
We present an ethical decision-making framework that refines a pre-trained reinforcement learning (RL) model using a task-agnostic ethical layer. Following initial training, the RL model undergoes ethical fine-tuning, wh…
Decision MakingEthicsLarge Language ModelNavigate+1MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning
Large language models are increasingly influencing human moral decisions, yet current approaches focus primarily on evaluating rather than actively steering their moral decisions. We formulate this as an out-of-distribut…
Reinforcement LearningMoral ScenariosVisual Distraction Undermines Moral Reasoning in Vision-Language Models
Moral reasoning is fundamental to safe Artificial Intelligence (AI), yet ensuring its consistency across modalities becomes critical as AI systems evolve from text-based assistants to embodied agents. Current safety tech…
Why should we ever automate moral decision making?
While people generally trust AI to make decisions in various aspects of their lives, concerns arise when AI is involved in decisions with significant moral implications. The absence of a precise mathematical framework fo…
Decision MakingEthicsLogical ReasoningActing for the Right Reasons: Creating Reason-Sensitive Artificial Moral Agents
We propose an extension of the reinforcement learning architecture that enables moral decision-making of reinforcement learning agents based on normative reasons. Central to this approach is a reason-based shield generat…
Decision Makingreinforcement-learningReinforcement Learning