Fabula Entropy Indexing: Objective Measures of Story Coherence
Automated story generation remains a difficult area of research because it lacks strong objective measures. Generated stories may be linguistically sound, but in many cases suffer poor narrative coherence required for a compelling, logically-sound story. To address this, we present Fabula Entropy Indexing (FEI), an evaluation method to assess story coherence by measuring the degree to which human participants agree with each other when answering true/false questions about stories. We devise two theoretically grounded measures of reader question-answering entropy, the entropy of world coherence (EWC), and the entropy of transitional coherence (ETC), focusing on global and local coherence, respectively. We evaluate these metrics by testing them on human-written stories and comparing against the same stories that have been corrupted to introduce incoherencies. We show that in these controlled studies, our entropy indices provide a reliable objective measure of story coherence.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringStory GenerationSimilar Papers 제목 키워드 기반
Narrative Variations in a Virtual Storyteller
Research on storytelling over the last 100 years has distinguished at least two levels of narrative representation (1) story, or fabula; and (2) discourse, or sujhet. We use this distinction to create Fabula Tales, a com…
Critical Confabulation: Can LLMs Hallucinate for Social Good?
LLMs hallucinate, yet some confabulations can have social affordances if carefully bounded. We propose critical confabulation (inspired by critical fabulation from literary and social theory), the use of LLM hallucinatio…
PersonaBank: A Corpus of Personal Narratives and Their Story Intention Graphs
We present a new corpus, PersonaBank, consisting of 108 personal stories from weblogs that have been annotated with their Story Intention Graphs, a deep representation of the fabula of a story. We describe the topics of …
FABULA: Intelligence Report Generation Using Retrieval-Augmented Narrative Construction
Narrative construction is the process of representing disparate event information into a logical plot structure that models an end to end story. Intelligence analysis is an example of a domain that can benefit tremendous…
DecoderLanguage ModellingLarge Language ModelRAG+2Using Referring Expression Generation to Model Literary Style
Novels and short stories are not just remarkable because of what events they represent. The narrative style they employ is significant. To understand the specific contributions of different aspects of this style, it is p…
modelReferring ExpressionReferring expression generation