Cross Modality 3D Navigation Using Reinforcement Learning and Neural Style Transfer
This paper presents the use of Multi-Agent Reinforcement Learning (MARL) to perform navigation in 3D anatomical volumes from medical imaging. We utilize Neural Style Transfer to create synthetic Computed Tomography (CT) agent gym environments and assess the generalization capabilities of our agents to clinical CT volumes. Our framework does not require any labelled clinical data and integrates easily with several image translation techniques, enabling cross modality applications. Further, we solely condition our agents on 2D slices, breaking grounds for 3D guidance in much more difficult imaging modalities, such as ultrasound imaging. This is an important step towards user guidance during the acquisition of standardised diagnostic view planes, improving diagnostic consistency and facilitating better case comparison.
Code (1)
Tasks
Computed Tomography (CT)DiagnosticMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Style TransferTranslationSimilar Papers 제목 키워드 기반
Multimodality-guided Image Style Transfer using Cross-modal GAN Inversion
Image Style Transfer (IST) is an interdisciplinary topic of computer vision and art that continuously attracts researchers' interests. Different from traditional Image-guided Image Style Transfer (IIST) methods that requ…
Style TransferSTEER: Unified Style Transfer with Expert Reinforcement
While text style transfer has many applications across natural language processing, the core premise of transferring from a single source style is unrealistic in a real-world setting. In this work, we focus on arbitrary …
Style TransferText Style TransferMIST GAN: Modality Imputation Using Style Transfer for MRI
MRI entails a great amount of cost, time and effort for the generation of all the modalities that are recommended for efficient diagnosis and treatment planning. Recent advancements in deep learning research show that ge…
DiversityImage GenerationImputationSSIM+1Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation
One of the most challenging topics in Natural Language Processing (NLP) is visually-grounded language understanding and reasoning. Outdoor vision-and-language navigation (VLN) is such a task where an agent follows natura…
Style TransferText Style TransferVision and Language NavigationCross-Modal Navigation with Multi-Agent Reinforcement Learning
Robust embodied navigation relies on complementary sensory cues. However, high-quality and well-aligned multi-modal data is often difficult to obtain in practice. Training a monolithic model is also challenging as rich m…
Multi-agent Reinforcement Learning