Ophora: A Large-Scale Data-Driven Text-Guided Ophthalmic Surgical Video Generation Model
In ophthalmic surgery, developing an AI system capable of interpreting surgical videos and predicting subsequent operations requires numerous ophthalmic surgical videos with high-quality annotations, which are difficult to collect due to privacy concerns and labor consumption. Text-guided video generation (T2V) emerges as a promising solution to overcome this issue by generating ophthalmic surgical videos based on surgeon instructions. In this paper, we present Ophora, a pioneering model that can generate ophthalmic surgical videos following natural language instructions. To construct Ophora, we first propose a Comprehensive Data Curation pipeline to convert narrative ophthalmic surgical videos into a large-scale, high-quality dataset comprising over 160K video-instruction pairs, Ophora-160K. Then, we propose a Progressive Video-Instruction Tuning scheme to transfer rich spatial-temporal knowledge from a T2V model pre-trained on natural video-text datasets for privacy-preserved ophthalmic surgical video generation based on Ophora-160K. Experiments on video quality evaluation via quantitative analysis and ophthalmologist feedback demonstrate that Ophora can generate realistic and reliable ophthalmic surgical videos based on surgeon instructions. We also validate the capability of Ophora for empowering downstream tasks of ophthalmic surgical workflow understanding. Code is available at https://github.com/mar-cry/Ophora.
Code (1)
Tasks
Video GenerationSimilar Papers 제목 키워드 기반
Take That for Me: Multimodal Exophora Resolution with Interactive Questioning for Ambiguous Out-of-View Instructions
Daily life support robots must interpret ambiguous verbal instructions involving demonstratives such as ``Bring me that cup,'' even when objects or users are out of the robot's view. Existing approaches to exophora resol…
Sound Source LocalizationJapanese Zero Reference Resolution Considering Exophora and Author/Reader Mentions
History of Onychophorology, 1826-2020
Velvet worms, or onychophorans, are animals of extraordinary importance in the study of evolution. This is the first history of their study. They were described by Lansdown Guilding (1797-1831). This paper identifies the…
AnatomyVelvet Worms (Onychophora) in Folklore and Art: Geographic Pattern, Types of Cultural Reference and Public Perception
Aims: To document and preserve folkloric beliefs and art inspired by velvet worms (Onychophora), rare invertebrates that are considered "living fossils", have full placental organs and capture prey with a rough "net" bui…
Cultural Vocal Bursts Intensity PredictionImage RetrievalThe Neoplasia as embryological phenomenon and its implication in the animal evolution and the origin of cancer. III. The role of flagellated cell fusion in the formation of the first animal and evolutionary clues to the Warburg effect
Cytasters have been underestimated in terms of their potential relevance to embryonic development and evolution. From the perspective discussed herein, structures such as the multiciliated cells of comb rows and balancer…