Get Your Model Puzzled: Introducing Crossword-Solving as a New NLP Benchmark
Solving crossword puzzles requires diverse reasoning capabilities, access to a vast amount of knowledge about language and the world, and the ability to satisfy the constraints imposed by the structure of the puzzle. In this work, we introduce solving crossword puzzles as a new natural language understanding task. We release a corpus of crossword puzzles collected from the New York Times daily crossword spanning 25 years and containing a total of 9152 puzzles, with an average of 85 clues per puzzle. These puzzles include a diverse set of clues: historic, factual, word meaning, synonyms/antonyms, fill-in-the-blank, abbreviations, prefixes/suffixes, wordplay, and cross-lingual, as well as clues that depend on the answers to other clues. We separately release the clue-answer pairs from these puzzles as an open-domain question answering dataset containing over half a million unique clue-answer pairs. For the question answering task, our baselines include several sequence-to-sequence and retrieval-based generative models. We also introduce a non-parametric constraint satisfaction baseline for solving the entire crossword puzzle. Finally, we propose an evaluation framework which consists of several complementary performance metrics.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language UnderstandingOpen-Domain Question AnsweringQuestion AnsweringRetrievalSimilar Papers 제목 키워드 기반
Down and Across: Introducing Crossword-Solving as a New NLP Benchmark
Solving crossword puzzles requires diverse reasoning capabilities, access to a vast amount of knowledge about language and the world, and the ability to satisfy the constraints imposed by the structure of the puzzle. In …
Natural Language UnderstandingOpen-Domain Question AnsweringQuestion AnsweringRetrievalPUZZLED: Jailbreaking LLMs through Word-Based Puzzles
As large language models (LLMs) are increasingly deployed across diverse domains, ensuring their safety has become a critical concern. In response, studies on jailbreak attacks have been actively growing. Existing approa…
Prompt EngineeringLanguage Models are Crossword Solvers
Crosswords are a form of word puzzle that require a solver to demonstrate a high degree of proficiency in natural language understanding, wordplay, reasoning, and world knowledge, along with adherence to character and le…
Natural Language UnderstandingWorld KnowledgeAutomated Crossword Solving
We present the Berkeley Crossword Solver, a state-of-the-art approach for automatically solving crossword puzzles. Our system works by generating answer candidates for each crossword clue using neural question answering …
Question AnsweringThe WebCrow French Crossword Solver
Crossword puzzles are one of the most popular word games, played in different languages all across the world, where riddle style can vary significantly from one country to another. Automated crossword resolution is chall…
Knowledge Graphs