Semiotically-grounded distant viewing of diagrams: insights from two multimodal corpora
In this article, we bring together theories of multimodal communication and computational methods to study how primary school science diagrams combine multiple expressive resources. We position our work within the field of digital humanities, and show how annotations informed by multimodality research, which target expressive resources and discourse structure, allow imposing structure on the output of computational methods. We illustrate our approach by analysing two multimodal diagram corpora: the first corpus is intended to support research on automatic diagram processing, whereas the second is oriented towards studying diagrams as a mode of communication. Our results show that multimodally-informed annotations can bring out structural patterns in the diagrams, which also extend across diagrams that deal with different topics.
Code (0)
등록된 구현이 없습니다.
Tasks
PositionVocal Bursts Valence PredictionSimilar Papers 제목 키워드 기반
Conspiracy Frame: a Semiotically-Driven Approach for Conspiracy Theories Detection
Conspiracy theories are anti-authoritarian narratives that lead to social conflict, impacting how people perceive political information. To help in understanding this issue, we introduce the Conspiracy Frame: a fine-grai…
On the Computational Entanglement of Distant Features in Adversarial Machine Learning
In this research, we introduce the concept of "computational entanglement," a phenomenon observed in overparameterized feedforward linear networks that enables the network to achieve zero loss by fitting random noise, ev…
GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagrams
Evaluating the symbolic reasoning of large language models (LLMs) calls for geometry benchmarks that require multi-step proofs grounded in both text and diagrams. However, existing benchmarks are often limited in scale a…
Semiotic Complexity and Its Epistemological Implications for Modeling Culture
Greater theorizing of methods in the computational humanities is needed for epistemological and interpretive clarity, and therefore the maturation of the field. In this paper, we frame such modeling work as engaging in t…
Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods
Despite their simple intuition, convolutions are more tedious to analyze than dense layers, which complicates the transfer of theoretical and algorithmic ideas to convolutions. We simplify convolutions by viewing them as…
Second-order methodsTensor Networks