Getting Reliable Annotations for Sarcasm in Online Dialogues
The language used in online forums differs in many ways from that of traditional language resources such as news. One difference is the use and frequency of nonliteral, subjective dialogue acts such as sarcasm. Whether the aim is to develop a theory of sarcasm in dialogue, or engineer automatic methods for reliably detecting sarcasm, a major challenge is simply the difficulty of getting enough reliably labelled examples. In this paper we describe our work on methods for achieving highly reliable sarcasm annotations from untrained annotators on Mechanical Turk. We explore the use of a number of common statistical reliability measures, such as Kappa, Karger's, Majority Class, and EM. We show that more sophisticated measures do not appear to yield better results for our data than simple measures such as assuming that the correct label is the one that a majority of Turkers apply.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Explainable Detection of Sarcasm in Social Media
Sarcasm is a linguistic expression often used to communicate the opposite of what is said, usually something that is very unpleasant with an intention to insult or ridicule. Inherent ambiguity in sarcastic expressions ma…
Sarcasm DetectionKoCoSa: Korean Context-aware Sarcasm Detection Dataset
Sarcasm is a way of verbal irony where someone says the opposite of what they mean, often to ridicule a person, situation, or idea. It is often difficult to detect sarcasm in the dialogue since detecting sarcasm should r…
Dataset GenerationSarcasm DetectionMuVaC: A Variational Causal Framework for Multimodal Sarcasm Understanding in Dialogues
The prevalence of sarcasm in multimodal dialogues on the social platforms presents a crucial yet challenging task for understanding the true intent behind online content. Comprehensive sarcasm analysis requires two key a…
Explanation GenerationSarcasm DetectionCausal InferenceGeneralizable Sarcasm Detection Is Just Around The Corner, Of Course!
We tested the robustness of sarcasm detection models by examining their behavior when fine-tuned on four sarcasm datasets containing varying characteristics of sarcasm: label source (authors vs. third-party), domain (soc…
Sarcasm DetectionCFMS: Towards Explainable and Fine-Grained Chinese Multimodal Sarcasm Detection Benchmark
Multimodal sarcasm detection has recently garnered significant attention. However, existing benchmarks suffer from coarse-grained annotations and limited cultural coverage, which hinder research into fine-grained semanti…
Explanation GenerationReinforcement LearningSarcasm Detection