From Detection to Anticipation: Online Understanding of Struggles across Various Tasks and Activities
Understanding human skill performance is essential for intelligent assistive systems, with struggle recognition offering a natural cue for identifying user difficulties. While prior work focuses on offline struggle classification and localization, real-time applications require models capable of detecting and anticipating struggle online. We reformulate struggle localization as an online detection task and further extend it to anticipation, predicting struggle moments before they occur. We adapt two off-the-shelf models as baselines for online struggle detection and anticipation. Online struggle detection achieves 70-80% per-frame mAP, while struggle anticipation up to 2 seconds ahead yields comparable performance with slight drops. We further examine generalization across tasks and activities and analyse the impact of skill evolution. Despite larger domain gaps in activity-level generalization, models still outperform random baselines by 4-20%. Our feature-based models run at up to 143 FPS, and the whole pipeline, including feature extraction, operates at around 20 FPS, sufficient for real-time assistive applications.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Memory-and-Anticipation Transformer for Online Action Understanding
Most existing forecasting systems are memory-based methods, which attempt to mimic human forecasting ability by employing various memory mechanisms and have progressed in temporal modeling for memory dependency. Neverthe…
Action DetectionAction UnderstandingOnline Action DetectionMANTA: Diffusion Mamba for Efficient and Effective Stochastic Long-Term Dense Anticipation
Our work addresses the problem of stochastic long-term dense anticipation. The goal of this task is to predict actions and their durations several minutes into the future based on provided video observations. Anticipatio…
MambaTemporally smooth online action detection using cycle-consistent future anticipation
Many video understanding tasks work in the offline setting by assuming that the input video is given from the start to the end. However, many real-world problems require the online setting, making a decision immediately …
Action DetectionAutonomous DrivingOnline Action DetectionVideo UnderstandingMANTA: Diffusion Mamba for Efficient and Effective Stochastic Long-Term Dense Action Anticipation
Long-term dense action anticipation is very challenging since it requires predicting actions and their durations several minutes into the future based on provided video observations. To model the uncertainty of futur…
Action AnticipationMambaJOADAA: joint online action detection and action anticipation
Action anticipation involves forecasting future actions by connecting past events to future ones. However, this reasoning ignores the real-life hierarchy of events which is considered to be composed of three main parts: …
Action AnticipationAction DetectionOnline Action Detection