Multimodal Shannon Game with Images
The Shannon game has long been used as a thought experiment in linguistics and NLP, asking participants to guess the next letter in a sentence based on its preceding context. We extend the game by introducing an optional extra modality in the form of image information. To investigate the impact of multimodal information in this game, we use human participants and a language model (LM, GPT-2). We show that the addition of image information improves both self-reported confidence and accuracy for both humans and LM. Certain word classes, such as nouns and determiners, benefit more from the additional modality information. The priming effect in both humans and the LM becomes more apparent as the context size (extra modality information + sentence context) increases. These findings highlight the potential of multimodal information in improving language understanding and modeling.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingSentenceSimilar Papers 제목 키워드 기반
A Comparative Genomic Analysis of Coronavirus Families Using Chaos Game Representation and Fisher-Shannon Complexity
From its first emergence in Wuhan, China in December, 2019 the COVID-19 pandemic has caused unprecedented health crisis throughout the world. The novel coronavirus disease is caused by severe acute respiratory syndrome c…
Information PlaneNemobot Games: Crafting Strategic AI Gaming Agents for Interactive Learning with Large Language Models
This paper introduces a new paradigm for AI game programming, leveraging large language models (LLMs) to extend and operationalize Claude Shannon's taxonomy of game-playing machines. Central to this paradigm is Nemobot, …
Reinforcement LearningMathematical ReasoningMultimodal Generative Learning Utilizing Jensen-Shannon-Divergence
Learning from different data types is a long-standing goal in machine learning research, as multiple information sources co-occur when describing natural phenomena. However, existing generative models that approximate a …
Saying the Unsaid: Revealing the Hidden Language of Multimodal Systems Through Telephone Games
Recent closed-source multimodal systems have made great advances, but their hidden language for understanding the world remains opaque because of their black-box architectures. In this paper, we use the systems' preferen…
The Entropy of Artificial Intelligence and a Case Study of AlphaZero from Shannon's Perspective
The recently released AlphaZero algorithm achieves superhuman performance in the games of chess, shogi and Go, which raises two open questions. Firstly, as there is a finite number of possibilities in the game, is there …
Reinforcement Learning