PoM: Efficient Image and Video Generation with the Polynomial Mixer

2024-11-19 · David Picard, Nicolas Dufour

Diffusion models based on Multi-Head Attention (MHA) have become ubiquitous to generate high quality images and videos. However, encoding an image or a video as a sequence of patches results in costly attention patterns, as the requirements both in terms of memory and compute grow quadratically. To alleviate this problem, we propose a drop-in replacement for MHA called the Polynomial Mixer (PoM) that has the benefit of encoding the entire sequence into an explicit state. PoM has a linear complexity with respect to the number of tokens. This explicit state also allows us to generate frames in a sequential fashion, minimizing memory and compute requirement, while still being able to train in parallel. We show the Polynomial Mixer is a universal sequence-to-sequence approximator, just like regular MHA. We adapt several Diffusion Transformers (DiT) for generating images and videos with PoM replacing MHA, and we obtain high quality samples while using less computational resources. The code is available at https://github.com/davidpicard/HoMM.

📄 PDF Abstract BibTeX arXiv:2411.12663

Code (1)

davidpicard/homm 공식 구현 pytorch

Tasks

Video Generation

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Multi-Head Attention 설명 없음

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

PoM: A Linear-Time Replacement for Attention with the Polynomial Mixer

2026-04-07 · David Picard, Nicolas Dufour, Lucas Degeorge, Arijit Ghosh 외 arxiv

This paper introduces the Polynomial Mixer (PoM), a novel token mixing mechanism with linear complexity that serves as a drop-in replacement for self-attention. PoM aggregates input tokens into a compact representation t…

Handwritten Text Recognition Image Generation Text Generation

ChebMixer: Efficient Graph Representation Learning with MLP Mixer

2024-03-25 · Xiaoyan Kui, Haonan Yan, Qinsong Li, Liming Chen 외

Graph neural networks have achieved remarkable success in learning graph representations, especially graph Transformer, which has recently shown superior performance on various graph mining tasks. However, graph Transfor…

Graph Mining Graph Representation Learning Image Segmentation Medical Image Segmentation+3

STS-Mixer: Spatio-Temporal-Spectral Mixer for 4D Point Cloud Video Understanding

2026-04-13 · Wenhao Li, Xueying Jiang, Gongjie Zhang, Xiaoqin Zhang 외 arxiv

4D point cloud videos capture rich spatial and temporal dynamics of scenes which possess unique values in various 4D understanding tasks. However, most existing methods work in the spatiotemporal domain where the underly…

Representation Learning 3D Action Recognition Semantic Segmentation

Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolation

2025-02-01 · Yang Cao, Zhao Song, Chiwun Yang

This paper considers an efficient video modeling process called Video Latent Flow Matching (VLFM). Unlike prior works, which randomly sampled latent patches for video generation, our method relies on current strong pre-t…

Image Generation Video Generation

Increasing diversity of omni-directional images generated from single image using cGAN based on MLPMixer

2023-09-15 · Atsuya Nakata, Ryuto Miyazaki, Takao Yamanaka

This paper proposes a novel approach to generating omni-directional images from a single snapshot picture. The previous method has relied on the generative adversarial networks based on convolutional neural networks (CNN…

Diversity Image Generation