Wordsyoudontknow: Evaluation of lexicon-based decompounding with unknown handling

CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models

While many languages possess processes of joining two or more words to create compound words, previous studies have been typically limited only to languages with excessively productive compound formation (e.g., German, D…

Using Semantics for Granularities of Tokenization

Depending on downstream applications, it is advisable to extend the notion of tokenization from low-level character-based token boundary detection to identification of meaningful and useful language units. This entails b…

Counting What Counts: Decompounding for Keyphrase Extraction

Corpus vs. Lexicon Supervision in Morphosyntactic Tagging: the Case of Slovene

In this paper we present a tagger developed for inflectionally rich languages for which both a training corpus and a lexicon are available. We do not constrain the tagger by the lexicon entries, allowing both for lexicon…

Paraphrase Types Elicit Prompt Engineering Capabilities

Much of the success of modern language models depends on finding a suitable prompt to instruct the model. Until now, it has been largely unknown how variations in the linguistic expression of prompts affect these models.…