RL-CSDia: Representation Learning of Computer Science Diagrams

2021-03-10 · Shaowei Wang, Lingling Zhang, Xuan Luo, Yi Yang, Xin Hu, Jun Liu

Recent studies on computer vision mainly focus on natural images that express real-world scenes. They achieve outstanding performance on diverse tasks such as visual question answering. Diagram is a special form of visual expression that frequently appears in the education field and is of great significance for learners to understand multimodal knowledge. Current research on diagrams preliminarily focuses on natural disciplines such as Biology and Geography, whose expressions are still similar to natural images. Another type of diagrams such as from Computer Science is composed of graphics containing complex topologies and relations, and research on this type of diagrams is still blank. The main challenges of graphic diagrams understanding are the rarity of data and the confusion of semantics, which are mainly reflected in the diversity of expressions. In this paper, we construct a novel dataset of graphic diagrams named Computer Science Diagrams (CSDia). It contains more than 1,200 diagrams and exhaustive annotations of objects and relations. Considering the visual noises caused by the various expressions in diagrams, we introduce the topology of diagrams to parse topological structure. After that, we propose Diagram Parsing Net (DPN) to represent the diagram from three branches: topology, visual feature, and text, and apply the model to the diagram classification task to evaluate the ability of diagrams understanding. The results show the effectiveness of the proposed DPN on diagrams understanding.

📄 PDF Abstract BibTeX arXiv:2103.05900

Code (0)

등록된 구현이 없습니다.

Tasks

Question Answering Representation Learning Visual Question Answering Visual Question Answering (VQA)

Methods 이 논문이 사용한 방법론

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Concatenated Skip Connection A Concatenated Skip Connection is a type of skip connection that seeks to reuse features by concatenating them to new layers, allowing more information to be retained from…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Max Pooling Max Pooling is a pooling operation that calculates the maximum value for patches of a feature map, and uses it to create a downsampled (pooled) feature map. It is usually…

Average Pooling 설명 없음

Residual Connection 설명 없음

Grouped Convolution A Grouped Convolution uses a group of convolutions - multiple kernels per layer - resulting in multiple channel outputs per layer. This leads to wider networks helping a…

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Text to Automata Diagrams: Comparing TikZ Code Generation with Direct Image Synthesis

2026-03-09 · Ethan Young, Zichun Wang, Aiden Taylor, Chance Jewell 외 arxiv

Diagrams are widely used in teaching computer science courses. They are useful in subjects such as automata and formal languages, data structures, etc. These diagrams, often drawn by students during exams or assignments,…

Code Generation

Unlocking UML Class Diagram Understanding in Vision Language Models

2026-05-12 · Artem Naboichenko, René Peinl arxiv

Although Vision Language Models (VLMs) have seen tremendous progress across all kinds of use cases, they still fall behind in answering questions regard-ing diagrams compared to photos. Although progress has been made in…

Visual Question Answering

Proactive Agentic Whiteboards: Enhancing Diagrammatic Learning

2025-12-01 · Suveen Ellawela, Sashenka Gamage, Dinithi Dissanayake arxiv

Educators frequently rely on diagrams to explain complex concepts during lectures, yet creating clear and complete visual representations in real time while simultaneously speaking can be cognitively demanding. Incomplet…

Code Completion

Classifying Diagrams and Their Parts using Graph Neural Networks: A Comparison of Crowd-Sourced and Expert Annotations

2019-12-05 · Tuomo Hiippala

This article compares two multimodal resources that consist of diagrams which describe topics in elementary school natural sciences. Both resources contain the same diagrams and represent their structure using graphs, bu…

Representation Learning

From Persistence to Survival: Hypothesis Testing, Effect Sizes and Vectorisation for Topological Features

2026-06-10 · Juliette Murris, Bernadette Stolz, Karsten Borgwardt arxiv

Persistence diagrams are common representations in topological data analysis, but they do not naturally live in a vector space, and the statistical tools developed for comparing them have largely evolved separately from …