Sparse Graph to Sequence Learning for Vision Conditioned Long Textual Sequence Generation

2020-07-12 · Aditya Mogadala, Marius Mosbach, Dietrich Klakow

Generating longer textual sequences when conditioned on the visual information is an interesting problem to explore. The challenge here proliferate over the standard vision conditioned sentence-level generation (e.g., image or video captioning) as it requires to produce a brief and coherent story describing the visual content. In this paper, we mask this Vision-to-Sequence as Graph-to-Sequence learning problem and approach it with the Transformer architecture. To be specific, we introduce Sparse Graph-to-Sequence Transformer (SGST) for encoding the graph and decoding a sequence. The encoder aims to directly encode graph-level semantics, while the decoder is used to generate longer sequences. Experiments conducted with the benchmark image paragraph dataset show that our proposed achieve 13.3% improvement on the CIDEr evaluation measure when comparing to the previous state-of-the-art approach.

📄 PDF Abstract BibTeX arXiv:2007.06077

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder Graph-to-SequenceSentenceVideo Captioning

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Multi-Head Attention 설명 없음

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Graph Conditioned Sparse-Attention for Improved Source Code Understanding

2021-12-01 · Junyan Cheng, Iordanis Fostiropoulos, Barry Boehm

Transformer architectures have been successfully used in learning source code representations. The fusion between a graph representation like Abstract Syntax Tree (AST) and a source code sequence makes the use of current…

Code Summarization Variable misuse

SAGE: Scene Graph-Aware Guidance and Execution for Long-Horizon Manipulation Tasks

2025-09-26 · Jialiang Li, Wenzheng Wu, Gaojing Zhang, Yifan Han 외 arxiv

Successfully solving long-horizon manipulation tasks remains a fundamental challenge. These tasks involve extended action sequences and complex object interactions, presenting a critical gap between high-level symbolic p…

Continuous Control Image Inpainting Image Editing

AI Choreographer: Music Conditioned 3D Dance Generation with AIST++

2021-01-21 · ICCV 2021 10 · RuiLong Li, Shan Yang, David A. Ross, Angjoo Kanazawa

We present AIST++, a new multi-modal dataset of 3D dance motion and music, along with FACT, a Full-Attention Cross-modal Transformer network for generating 3D dance motion conditioned on music. The proposed AIST++ datase…

Motion Generation Motion Synthesis Pose Estimation

Sparse Instance Conditioned Multimodal Trajectory Prediction

2023-01-01 · ICCV 2023 1 · Yonghao Dong, Le Wang, Sanping Zhou, Gang Hua

Pedestrian trajectory prediction is critical in many vision tasks but challenging due to the multimodality of the future trajectory. Most existing methods predict multimodal trajectories conditioned by goals (future …

Future prediction Pedestrian Trajectory Prediction Prediction Trajectory Prediction

Action-conditioned video data improves predictability

2024-04-08 · Meenakshi Sarkar, Debasish Ghose

Long-term video generation and prediction remain challenging tasks in computer vision, particularly in partially observable scenarios where cameras are mounted on moving platforms. The interaction between observed image …

Video Generation