CapsDT: Diffusion-Transformer for Capsule Robot Manipulation

2025-06-19 · Xiting He, Mingwu Su, Xinqi Jiang, Long Bai, JIEWEN LAI, Hongliang Ren

Vision-Language-Action (VLA) models have emerged as a prominent research area, showcasing significant potential across a variety of applications. However, their performance in endoscopy robotics, particularly endoscopy capsule robots that perform actions within the digestive system, remains unexplored. The integration of VLA models into endoscopy robots allows more intuitive and efficient interactions between human operators and medical devices, improving both diagnostic accuracy and treatment outcomes. In this work, we design CapsDT, a Diffusion Transformer model for capsule robot manipulation in the stomach. By processing interleaved visual inputs, and textual instructions, CapsDT can infer corresponding robotic control signals to facilitate endoscopy tasks. In addition, we developed a capsule endoscopy robot system, a capsule robot controlled by a robotic arm-held magnet, addressing different levels of four endoscopy tasks and creating corresponding capsule robot datasets within the stomach simulator. Comprehensive evaluations on various robotic tasks indicate that CapsDT can serve as a robust vision-language generalist, achieving state-of-the-art performance in various levels of endoscopy tasks while achieving a 26.25% success rate in real-world simulation manipulation.

📄 PDF Abstract BibTeX arXiv:2506.16263

Code (0)

등록된 구현이 없습니다.

Tasks

DiagnosticRobot ManipulationVision-Language-Action

Methods 이 논문이 사용한 방법론

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Transformer A Transformer is a model architecture that eschews recurrence and instead relies entirely on an [attention…

Safe Robotic Capsule Cleaning with Integrated Transpupillary and Intraocular Optical Coherence Tomography

2025-07-18 · Yu-Ting Lai, Yasamin Foroutani, Aya Barzelay, Tsu-Chin Tsao arxiv

Secondary cataract is one of the most common complications of vision loss due to the proliferation of residual lens materials that naturally grow on the lens capsule after cataract surgery. A potential treatment is capsu…

U-DiT Policy: U-shaped Diffusion Transformers for Robotic Manipulation

2025-09-29 · Linzhi Wu, Aoran Mei, Xiyue Wang, Guo-Niu Zhu 외 arxiv

Diffusion-based methods have been acknowledged as a powerful paradigm for end-to-end visuomotor control in robotics. Most existing approaches adopt a Diffusion Policy in U-Net architecture (DP-U), which, while effective,…

RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation

2024-10-10 · Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan 외

Bimanual manipulation is essential in robotics, yet developing foundation models is extremely challenging due to the inherent complexity of coordinating two robot arms (leading to multi-modal action distributions) and th…

Zero-shot Generalization

Tenma: Robust Cross-Embodiment Robot Manipulation with Diffusion Transformer

2025-09-15 · Travis Davies, Yiqi Huang, Yunxin Liu, Xiang Chen 외 arxiv

Scaling Transformer policies and diffusion models has advanced robotic manipulation, yet combining these techniques in lightweight, cross-embodiment learning settings remains challenging. We study design choices that mos…

Robot Manipulation

H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation

2025-07-31 · Hongzhe Bi, Lingxuan Wu, Tianwei Lin, Hengkai Tan 외 arxiv

Imitation learning for robotic manipulation faces a fundamental challenge: the scarcity of large-scale, high-quality robot demonstration data. Recent robotic foundation models often pre-train on cross-embodiment robot da…

Robot Manipulation Few-Shot Learning