MATILDA - Multi-AnnoTator multi-language InteractiveLight-weight Dialogue Annotator
Dialogue Systems are becoming ubiquitous in various forms and shapes - virtual assistants(Siri, Alexa, etc.), chat-bots, customer sup-port, chit-chat systems just to name a few.The advances in language models and their publication have democratised advanced NLP.However, data remains a crucial bottleneck.Our contribution to this essential pillar isMATILDA, to the best of our knowledge the first multi-annotator, multi-language dialogue annotation tool. MATILDA allows the creation of corpora, the management of users, the annotation of dialogues, the quick adaptation of the user interface to any language and the resolution of inter-annotator disagreement. We evaluate the tool on ease of use, annotation speed and interannotation resolution for both experts and novices and conclude that this tool not only supports the full pipeline for dialogue annotation, but also allows non-technical people to easily use it. We are completely open-sourcing the tool at https://github.com/wluper/matilda and provide a tutorial video1.
Code (1)
Tasks
ManagementSimilar Papers 제목 키워드 기반
Matilda: Engine-Agnostic Search with Human Policy Guidance
Chess engines have evolved from search-based systems optimized for strength to neural policies optimized for predicting human decisions. Existing approaches largely separate these goals: search engines achieve superhuman…
Representation LearningDravidianMultiModality: A Dataset for Multi-modal Sentiment Analysis in Tamil and Malayalam
Human communication is inherently multimodal and asynchronous. Analyzing human emotions and sentiment is an emerging field of artificial intelligence. We are witnessing an increasing amount of multimodal content in local…
Multimodal Sentiment AnalysisSentiment AnalysisBeyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks
Disagreement in annotation is a common phenomenon in the development of NLP datasets and serves as a valuable source of insight. While majority voting remains the dominant strategy for aggregating labels, recent work has…
Emotion ClassificationHate Speech DetectionSentiment AnalysisYou Are What You Annotate: Towards Better Models through Annotator Representations
Annotator disagreement is ubiquitous in natural language processing (NLP) tasks. There are multiple reasons for such disagreements, including the subjectivity of the task, difficult cases, unclear guidelines, and so on. …
Bias in, Bias out: Annotation Bias in Multilingual Large Language Models
Annotation bias in NLP datasets remains a major challenge for developing multilingual Large Language Models (LLMs), particularly in culturally diverse settings. Bias from task framing, annotator subjectivity, and cultura…