SegmATRon: Embodied Adaptive Semantic Segmentation for Indoor Environment
This paper presents an adaptive transformer model named SegmATRon for embodied image semantic segmentation. Its distinctive feature is the adaptation of model weights during inference on several images using a hybrid multicomponent loss function. We studied this model on datasets collected in the photorealistic Habitat and the synthetic AI2-THOR Simulators. We showed that obtaining additional images using the agent's actions in an indoor environment can improve the quality of semantic segmentation. The code of the proposed approach and datasets are publicly available at https://github.com/wingrune/SegmATRon.
Code (1)
Tasks
SegmentationSemantic SegmentationSimilar Papers 제목 키워드 기반
The Replica Dataset: A Digital Replica of Indoor Spaces
We introduce Replica, a dataset of 18 highly photo-realistic 3D indoor scene reconstructions at room and building scale. Each scene consists of a dense mesh, high-resolution high-dynamic-range (HDR) textures, per-primiti…
3D Scene ReconstructionInstruction FollowingQuestion AnsweringSemantic SegmentationJSMNet Improving Indoor Point Cloud Semantic and Instance Segmentation through Self-Attention and Multiscale
The semantic understanding of indoor 3D point cloud data is crucial for a range of subsequent applications, including indoor service robots, navigation systems, and digital twin engineering. Global features are crucial f…
Instance SegmentationSegmentationSemantic SegmentationSemantic Mapping in Indoor Embodied AI -- A Survey on Advances, Challenges, and Future Directions
Intelligent embodied agents (e.g. robots) need to perform complex semantic tasks in unfamiliar environments. Among many skills that the agents need to possess, building and maintaining a semantic map of the environment i…
Embodied Amodal Recognition: Learning to Move to Perceive Objects
Passive visual systems typically fail to recognize objects in the amodal setting where they are heavily occluded. In contrast, humans and other embodied agents have the ability to move in the environment and actively con…
ObjectObject LocalizationSemantic SegmentationEmbodied Visual Active Learning for Semantic Segmentation
We study the task of embodied visual active learning, where an agent is set to explore a 3d environment with the goal to acquire visual scene understanding by actively selecting views for which to request annotation. Whi…
Active LearningDeep Reinforcement LearningScene UnderstandingSemantic Segmentation