Simultaneously-Collected Multimodal Lying Pose Dataset: Towards In-Bed Human Pose Monitoring under Adverse Vision Conditions

2020-08-20 · Shuangjun Liu, Xiaofei Huang, Nihang Fu, Cheng Li, Zhongnan Su, Sarah Ostadabbas

Computer vision (CV) has achieved great success in interpreting semantic meanings from images, yet CV algorithms can be brittle for tasks with adverse vision conditions and the ones suffering from data/label pair limitation. One of this tasks is in-bed human pose estimation, which has significant values in many healthcare applications. In-bed pose monitoring in natural settings could involve complete darkness or full occlusion. Furthermore, the lack of publicly available in-bed pose datasets hinders the use of many successful pose estimation algorithms for this task. In this paper, we introduce our Simultaneously-collected multimodal Lying Pose (SLP) dataset, which includes in-bed pose images from 109 participants captured using multiple imaging modalities including RGB, long wave infrared, depth, and pressure map. We also present a physical hyper parameter tuning strategy for ground truth pose label generation under extreme conditions such as lights off and being fully covered by a sheet/blanket. SLP design is compatible with the mainstream human pose datasets, therefore, the state-of-the-art 2D pose estimation models can be trained effectively with SLP data with promising performance as high as 95% at PCKh@0.5 on a single modality. The pose estimation performance can be further improved by including additional modalities through collaboration.

📄 PDF Abstract BibTeX arXiv:2008.08735

Code (2)

ostadabbas/SLP-Dataset-and-Code 공식 구현 pytorch

hirokichi-k/extended_SLPdataset

Tasks

2D Pose Estimation Pose Estimation

Under the Cover Infant Pose Estimation using Multimodal Data

2022-10-03 · Daniel G. Kyrollos, Anthony Fuller, Kim Greenwood, JoAnn Harrold 외

Infant pose monitoring during sleep has multiple applications in both healthcare and home settings. In a healthcare setting, pose detection can be used for region of interest detection and movement detection for nonconta…

Pose Estimation Transfer Learning

ARMOR v0.1: Empowering Autoregressive Multimodal Understanding Model with Interleaved Multimodal Generation via Asymmetric Synergy

2025-03-09 · Jianwen Sun, Yukang Feng, Chuanhao Li, Fanrui Zhang 외

Unified models (UniMs) for multimodal understanding and generation have recently received much attention in the area of vision and language. Existing UniMs are designed to simultaneously learn both multimodal understandi…

Decoder Image Generation multimodal generation

Lightweight Multimodal Adaptation of Vision Language Models for Species Recognition and Habitat Context Interpretation in Drone Thermal Imagery

2026-04-07 · Hao Chen, Fang Qiu, Fangchao Dong, Defei Yang 외 arxiv

This study proposes a lightweight multimodal adaptation framework to bridge the representation gap between RGB-pretrained VLMs and thermal infrared imagery, and demonstrates its practical utility using a real drone-colle…

Exploring Text Specific and Blackbox Fairness Algorithms in Multimodal Clinical NLP

2020-11-19 · EMNLP (ClinicalNLP) 2020 11 · John Chen, Ian Berlot-Attwell, Safwan Hossain, Xindi Wang 외

Clinical machine learning is increasingly multimodal, collected in both structured tabular formats and unstructured forms such as freetext. We propose a novel task of exploring fairness on a multimodal clinical dataset, …

Fairness Word Embeddings

Latent Multimodal Functional Graphical Model Estimation

2022-10-31 · Katherine Tsai, Boxin Zhao, Sanmi Koyejo, Mladen Kolar

Joint multimodal functional data acquisition, where functional data from multiple modes are measured simultaneously from the same subject, has emerged as an exciting modern approach enabled by recent engineering breakthr…

Functional Connectivity model