Similarity or deeper understanding? Analyzing the TED-Q dataset of evoked questions
We take a close look at a recent dataset of TED-talks annotated with the questions they implicitly evoke, TED-Q (Westera et al., 2020). We test to what extent the relation between a discourse and the questions it evokes is merely one of similarity or association, as opposed to deeper semantic/pragmatic interpretation. We do so by turning the TED-Q dataset into a binary classification task, constructing an analogous task from explicit questions we extract from the BookCorpus (Zhu et al., 2015), and fitting a BERT-based classifier alongside models based on different notions of similarity. The BERT-based classifier, achieving close to human performance, outperforms all similarity-based models, suggesting that there is more to identifying true evoked questions than plain similarity.
Code (1)
Tasks
Binary ClassificationSimilar Papers 제목 키워드 기반
Analyzing CodeBERT's Performance on Natural Language Code Search
Large language models such as CodeBERT perform very well on tasks such as natural language code search. We show that this is most likely due to the high token overlap and similarity between the queries and the code in da…
Code SearchAnalyzing Continuous Semantic Shifts with Diachronic Word Similarity Matrices
The meanings and relationships of words shift over time. This phenomenon is referred to as semantic shift. Research focused on understanding how semantic shifts occur over multiple time periods is essential for gaining a…
Word EmbeddingsWord SimilarityPragmatic Frames Evoked by Gestures: A FrameNet Brasil Approach to Multimodality in Turn Organization
This paper proposes a framework for modeling multimodal conversational turn organization via the proposition of correlations between language and interactive gestures, based on analysis as to how pragmatic frames are con…
Analyzing the Brain's Dynamic Response to Targeted Stimulation using Generative Modeling
Generative models of brain activity have been instrumental in testing hypothesized mechanisms underlying brain dynamics against experimental datasets. Beyond capturing the key mechanisms underlying spontaneous brain dyna…
Explaining Text Similarity in Transformer Models
As Transformers have become state-of-the-art models for natural language processing (NLP) tasks, the need to understand and explain their predictions is increasingly apparent. Especially in unsupervised applications, suc…
Information RetrievalRetrievalSemantic SimilaritySemantic Textual Similarity+2