Procedural Text Generation from an Execution Video

2017-11-01 · IJCNLP 2017 11 · Atsushi Ushiku, Hayato Hashimoto, Atsushi Hashimoto, Shinsuke Mori

In recent years, there has been a surge of interest in automatically describing images or videos in a natural language. These descriptions are useful for image/video search, etc. In this paper, we focus on procedure execution videos, in which a human makes or repairs something and propose a method for generating procedural texts from them. Since video/text pairs available are limited in size, the direct application of end-to-end deep learning is not feasible. Thus we propose to train Faster R-CNN network for object recognition and LSTM for text generation and combine them at run time. We took pairs of recipe and cooking video, generated a recipe from a video, and compared it with the original recipe. The experimental results showed that our method can produce a recipe as accurate as the state-of-the-art scene descriptions.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Object Recognition Text Generation Video Captioning

Methods 이 논문이 사용한 방법론

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

RPN A Region Proposal Network, or RPN, is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

RoIPool 설명 없음

Faster R-CNN Faster R-CNN is an object detection model that improves on Fast R-CNN by utilising a region proposal network…

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting

2024-12-16 · Muhammet Furkan Ilaslan, Ali Koksal, Kevin Qinhong Lin, Burak Satar 외

Large Language Model (LLM)-based agents have shown promise in procedural tasks, but the potential of multimodal instructions augmented by texts and videos to assist users remains under-explored. To address this gap, we p…

Informativeness Large Language Model Text Generation Text-to-Video Generation+2

RECIPE: Procedural Planning via Grounding in Instructional Video

2026-05-19 · Luigi Seminara, Antonino Furnari, Lorenzo Torresani arxiv

Visual planning asks a model to generate the remaining steps of a procedure in natural language given a partial video context and a goal. Progress on this task is bottlenecked by annotation: clean labeled datasets are sm…

SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agents

2026-03-09 · Yu Yang, Yue Liao, Jianbiao Mei, Baisen Wang 외 arxiv

Long-horizon action-conditioned video generation aims to synthesize temporally coherent videos that follow complex action instructions over extended horizons, requiring procedural ordering, persistent action execution, a…

Video Generation

ProSkill: Segment-Level Skill Assessment in Procedural Videos

2026-01-28 · Michele Mazzamuto, Daniele Di Mauro, Gianpiero Francesca, Giovanni Maria Farinella 외 arxiv

Skill assessment in procedural videos is crucial for the objective evaluation of human performance in settings such as manufacturing and procedural daily tasks. Current research on skill assessment has predominantly focu…

How to Correctly Make Mistakes: A Framework for Constructing and Benchmarking Mistake Aware Egocentric Procedural Videos

2026-04-16 · Olga Loginova, Frank Keller arxiv

Reliable procedural monitoring in video requires exposure to naturally occurring human errors and the recoveries that follow. In egocentric recordings, mistakes are often partially occluded by hands and revealed through …