Diffusion-Based Action Recognition Generalizes to Untrained Domains

2025-09-10 · Rogerio Guimaraes, Frank Xiao, Pietro Perona, Markus Marks arxiv

Humans can recognize the same actions despite large context and viewpoint variations, such as differences between species (walking in spiders vs. horses), viewpoints (egocentric vs. third-person), and contexts (real life vs movies). Current deep learning models struggle with such generalization. We propose using features generated by a Vision Diffusion Model (VDM), aggregated via a transformer, to achieve human-like action recognition across these challenging conditions. We find that generalization is enhanced by the use of a model conditioned on earlier timesteps of the diffusion process to highlight semantic information over pixel level details in the extracted features. We experimentally explore the generalization properties of our approach in classifying actions across animal species, across different viewing angles, and different recording contexts. Our model sets a new state-of-the-art across all three generalization benchmarks, bringing machine action recognition closer to human-like robustness. Project page: https://www.vision.caltech.edu/actiondiff. Code: https://github.com/frankyaoxiao/ActionDiff

📄 PDF Abstract BibTeX arXiv:2509.08908

Code (0)

등록된 구현이 없습니다.

Tasks

Action Recognition

An Underparametrized Deep Decoder Architecture for Graph Signals

2019-08-02 · Samuel Rey, Antonio G. Marques, Santiago Segarra

While deep convolutional architectures have achieved remarkable results in a gamut of supervised applications dealing with images and speech, recent works show that deep untrained non-convolutional architectures can also…

Clustering Decoder Denoising Image Compression

Conceptual Design of Human-Drone Communication in Collaborative Environments

2020-04-30 · Hans Dermot Doran, Monika Reif, Marco Oehler, Curdin Stoehr 외

Autonomous robots and drones will work collaboratively and cooperatively in tomorrow's industry and agriculture. Before this becomes a reality, some form of standardised communication between man and machine must be esta…

Effective Data Augmentation With Diffusion Models

2023-02-07 · Brandon Trabucco, Kyle Doherty, Max Gurinas, Ruslan Salakhutdinov

Data augmentation is one of the most prevalent tools in deep learning, underpinning many recent advances, including those from classification, generative models, and representation learning. The standard approach to data…

Data Augmentation Diversity Few-Shot Image Classification image-classification+2

Label Efficient Learning of Transferable Representations across Domains and Tasks

2017-11-30 · NeurIPS 2017 · Zelun Luo, Yuliang Zou, Judy Hoffman, Li Fei-Fei

We propose a framework that learns a representation transferable across different domains and tasks in a label efficient manner. Our approach battles domain shift with a domain adversarial loss, and generalizes the embed…

Action Recognition Metric Learning Object Recognition Temporal Action Localization+1

Label Efficient Learning of Transferable Representations acrosss Domains and Tasks

2017-12-01 · NeurIPS 2017 12 · Zelun Luo, Yuliang Zou, Judy Hoffman, Li F. Fei-Fei

We propose a framework that learns a representation transferable across different domains and tasks in a data efficient manner. Our approach battles domain shift with a domain adversarial loss, and generalizes the embedd…