MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving

2023-03-15 · CVPR 2023 1 · Jiale Li, Hang Dai, Hao Han, Yong Ding

LiDAR and camera are two modalities available for 3D semantic segmentation in autonomous driving. The popular LiDAR-only methods severely suffer from inferior segmentation on small and distant objects due to insufficient laser points, while the robust multi-modal solution is under-explored, where we investigate three crucial inherent difficulties: modality heterogeneity, limited sensor field of view intersection, and multi-modal data augmentation. We propose a multi-modal 3D semantic segmentation model (MSeg3D) with joint intra-modal feature extraction and inter-modal feature fusion to mitigate the modality heterogeneity. The multi-modal fusion in MSeg3D consists of geometry-based feature fusion GF-Phase, cross-modal feature completion, and semantic-based feature fusion SF-Phase on all visible points. The multi-modal data augmentation is reinvigorated by applying asymmetric transformations on LiDAR point cloud and multi-camera images individually, which benefits the model training with diversified augmentation transformations. MSeg3D achieves state-of-the-art results on nuScenes, Waymo, and SemanticKITTI datasets. Under the malfunctioning multi-camera input and the multi-frame point clouds input, MSeg3D still shows robustness and improves the LiDAR-only baseline. Our code is publicly available at \url{https://github.com/jialeli1/lidarseg3d}.

📄 PDF Abstract BibTeX arXiv:2303.08600

Code (1)

jialeli1/lidarseg3d 공식 구현 pytorch

Tasks

3D Semantic Segmentation Autonomous Driving Data Augmentation Segmentation Semantic Segmentation

HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS

2021-01-18 · Chien-Hsiang Huang, Hung-Yu Wu, Youn-Long Lin

We propose a new convolution neural network called HarDNet-MSEG for polyp segmentation. It achieves SOTA in both accuracy and inference speed on five popular datasets. For Kvasir-SEG, HarDNet-MSEG delivers 0.904 mean Dic…

Decoder GPU image-classification Image Classification+9

GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models

2022-10-05 · Chen Liang, Wenguan Wang, Jiaxu Miao, Yi Yang

Prevalent semantic segmentation solutions are, in essence, a dense discriminative classifier of p(class|pixel feature). Though straightforward, this de facto paradigm neglects the underlying data distribution p(pixel fea…

Out-of-Distribution Detection Segmentation Semantic Segmentation

MSEG-VCUQ: Multimodal SEGmentation with Enhanced Vision Foundation Models, Convolutional Neural Networks, and Uncertainty Quantification for High-Speed Video Phase Detection Data

2024-11-12 · Chika Maduabuchi, Ericmoore Jossou, Matteo Bucci

High-speed video (HSV) phase detection (PD) segmentation is crucial for monitoring vapor, liquid, and microlayer phases in industrial processes. While CNN-based models like U-Net have shown success in simplified shadowgr…

Segmentation Uncertainty Quantification Video Segmentation Video Semantic Segmentation

MSeg: A Composite Dataset for Multi-domain Semantic Segmentation

2021-12-27 · CVPR 2020 6 · John Lambert, Zhuang Liu, Ozan Sener, James Hays 외

We present MSeg, a composite dataset that unifies semantic segmentation datasets from different domains. A naive merge of the constituent datasets yields poor performance due to inconsistent taxonomies and annotation pra…

Computational Efficiency Instance Segmentation Panoptic Segmentation Scene Understanding+2

A Simple Framework for Text-Supervised Semantic Segmentation

2023-01-01 · CVPR 2023 1 · Muyang Yi, Quan Cui, Hao Wu, Cheng Yang 외

Text-supervised semantic segmentation is a novel research topic that allows semantic segments to emerge with image-text contrasting. However, pioneering methods could be subject to specifically designed network archi…

Segmentation Semantic Segmentation