Object Detection for Understanding Assembly Instruction Using Context-aware Data Augmentation and Cascade Mask R-CNN

2021-01-07 · Joosoon Lee, Seongju Lee, Seunghyeok Back, Sungho Shin, Kyoobin Lee

Understanding assembly instruction has the potential to enhance the robot s task planning ability and enables advanced robotic applications. To recognize the key components from the 2D assembly instruction image, We mainly focus on segmenting the speech bubble area, which contains lots of information about instructions. For this, We applied Cascade Mask R-CNN and developed a context-aware data augmentation scheme for speech bubble segmentation, which randomly combines images cuts by considering the context of assembly instructions. We showed that the proposed augmentation scheme achieves a better segmentation performance compared to the existing augmentation algorithm by increasing the diversity of trainable data while considering the distribution of components locations. Also, we showed that deep learning can be useful to understand assembly instruction by detecting the essential objects in the assembly instruction, such as tools and parts.

📄 PDF Abstract BibTeX arXiv:2101.02509

Code (0)

등록된 구현이 없습니다.

Tasks

Data AugmentationDiversityobject-detection Object Detection SegmentationTask Planning

Methods 이 논문이 사용한 방법론

RPN A Region Proposal Network, or RPN, is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

RoIAlign Region of Interest Align, or RoIAlign, is an operation for extracting a small feature map from each RoI in detection and segmentation based tasks. It removes the harsh…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Cascade Mask R-CNN Cascade Mask R-CNN extends Cascade R-CNN to instance segmentation, by adding a mask head to the cascade. In the [Mask…

Mask R-CNN Mask R-CNN extends Faster R-CNN to solve instance segmentation tasks. It achieves this by adding a branch for predicting an…

LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants

2025-07-07 · Haochen Huang, Jiahuan Pei, Mohammad Aliannejadi, Xin Sun 외 arxiv

Vision-language models (VLMs) are facing the challenges of understanding and following multimodal assembly instructions, particularly when fine-grained spatial reasoning and precise object state detection are required. I…

Spatial Reasoning Object Detection

AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects

2026-05-13 · Danrui Li, Jiahao Zhang, Bernhard Egger, Moitreya Chatterjee 외 arxiv

Assembling objects from parts requires understanding multimodal instructions, linking them to 3D components, and predicting physically plausible 6-DoF motions for each assembly step. Existing datasets focus on simplified…

Pose Estimation

IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos

2024-11-18 · Yunong Liu, Cristobal Eyzaguirre, Manling Li, Shubh Khanna 외

Shape assembly is a ubiquitous task in daily life, integral for constructing complex 3D structures like IKEA furniture. While significant progress has been made in developing autonomous agents for shape assembly, existin…

Pose Estimation Semantic Segmentation Video Object Segmentation Video Semantic Segmentation

BinBert: Binary Code Understanding with a Fine-tunable and Execution-aware Transformer

2022-08-13 · Fiorella Artuso, Marco Mormando, Giuseppe A. Di Luna, Leonardo Querzoni

A recent trend in binary code analysis promotes the use of neural solutions based on instruction embedding models. An instruction embedding model is a neural network that transforms sequences of assembly instructions int…

General Knowledge

Generative Timelines for Instructed Visual Assembly

2024-11-19 · Alejandro Pardo, Jui-Hsien Wang, Bernard Ghanem, Josef Sivic 외

The objective of this work is to manipulate visual timelines (e.g. a video) through natural language instructions, making complex timeline editing tasks accessible to non-expert or potentially even disabled users. We cal…

Language Modelling