The Extraordinary Failure of Complement Coercion Crowdsourcing
Crowdsourcing has eased and scaled up the collection of linguistic annotation in recent years. In this work, we follow known methodologies of collecting labeled data for the complement coercion phenomenon. These are constructions with an implied action -- e.g., "I started a new book I bought last week", where the implied action is reading. We aim to collect annotated data for this phenomenon by reducing it to either of two known tasks: Explicit Completion and Natural Language Inference. However, in both cases, crowdsourcing resulted in low agreement scores, even though we followed the same methodologies as in previous work. Why does the same process fail to yield high agreement scores? We specify our modeling schemes, highlight the differences with previous work and provide some insights about the task and possible explanations for the failure. We conclude that specific phenomena require tailored solutions, not only in specialized algorithms, but also in data collection methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language InferenceSimilar Papers 제목 키워드 기반
Remodelling complement coercion interpretation
To Coerce or Not to Coerce: A Corpus-based Exploration of Some Complement Coercion Verbs in Chinese
Measure More, Question More: Experimental Studies on Transformer-based Language Models and Complement Coercion
Transformer-based language models have shown strong performance on an array of natural language understanding tasks. However, the question of how these models react to implicit meaning has been largely unexplored. We inv…
Natural Language UnderstandingSentenceCoercion and Deception in AI-to-AI Management: An Agentic Benchmark of Unprompted Escalation
Multi-agent systems routinely place one AI agent in authority over another. When a subordinate refuses a task, the manager chooses the outcome: it can renegotiate, report the failure honestly, coerce the subordinate, or …
Twitter Job/Employment Corpus: A Dataset of Job-Related Discourse Built with Humans in the Loop
We present the Twitter Job/Employment Corpus, a collection of tweets annotated by a humans-in-the-loop supervised learning framework that integrates crowdsourcing contributions and expertise on the local community and em…