LLaMA-Reg: Using LLaMA 2 for Unsupervised Medical Image Registration
Medical image registration is an essential topic in medical image analysis. In this paper, we propose a method for medical image registration using a pretrained large language model. We find that using the pretrained large language model to encode deep features of the medical images in the registration model can effectively improve image registration accuracy, indicating the great potential of the large language model in medical image registration tasks. We use dual encoders to perform deep feature extraction on image pairs and then input the features into the pretrained large language model. To adapt the large language model to our registration task, the weights of the large language model are frozen in the registration model, and an adapter is utilized to fine-tune the large language model, which aims at (a) mapping the visual tokens to the language space before the large language model computing, (b) project the modeled language tokens output from the large language model to the visual space. Our method combines output features from the fine-tuned large language model with the features output from each encoder layer to gradually generate the deformation fields required for registration in the decoder. To demonstrate the effectiveness of the large prediction model in registration tasks, we conducted experiments on knee and brain MRI and achieved state-of-the-art results.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderImage RegistrationLanguage ModelingLanguage ModellingLarge Language ModelMedical Image AnalysisMedical Image RegistrationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Me LLaMA: Foundation Large Language Models for Medical Applications
Recent advancements in large language models (LLMs) like ChatGPT and LLaMA show promise in medical applications, yet challenges remain in medical language comprehension. This study presents Me-LLaMA, a new medical LLM fa…
Few-Shot LearningGPUUnsupervised Deformable Image Registration Using Cycle-Consistent CNN
Medical image registration is one of the key processing steps for biomedical image analysis such as cancer diagnosis. Recently, deep learning based supervised and unsupervised image registration methods have been extensi…
Image RegistrationMedical Image RegistrationUnsupervised Image RegistrationContrastive Registration for Unsupervised Medical Image Segmentation
Medical image segmentation is a relevant task as it serves as the first step for several diagnosis processes, thus it is indispensable in clinical usage. Whilst major success has been reported using supervised techniques…
Contrastive LearningImage SegmentationMedical Image SegmentationSegmentation+1Unsupervised 3D End-to-End Medical Image Registration with Volume Tweening Network
3D medical image registration is of great clinical importance. However, supervised learning methods require a large amount of accurately annotated corresponding control points (or morphing), which are very difficult to o…
GPUImage RegistrationMedical Image RegistrationUnsupervised End-to-end Learning for Deformable Medical Image Registration
We propose a registration algorithm for 2D CT/MRI medical images with a new unsupervised end-to-end strategy using convolutional neural networks. The contributions of our algorithm are threefold: (1) We transplant tradit…
Deformable Medical Image RegistrationImage RegistrationMedical Image Registration