Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
What are the units of text that we want to model? From bytes to multi-word expressions, text can be analyzed and generated at many granularities. Until recently, most natural language processing (NLP) models operated over words, treating those as discrete and atomic tokens, but starting with byte-pair encoding (BPE), subword-based approaches have become dominant in many areas, enabling small vocabularies while still allowing for fast inference. Is the end of the road character-level model or byte-level processing? In this survey, we connect several lines of work from the pre-neural and neural era, by showing how hybrid approaches of words and characters as well as subword-based approaches based on learned segmentation have been proposed and evaluated. We conclude that there is and likely will never be a silver bullet singular solution for all applications and that thinking seriously about tokenization remains important for many applications.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Tragic and Comical Networks. Clustering Dramatic Genres According to Structural Properties
There is a growing tradition in the joint field of network studies and drama history that produces interpretations from the character networks of the plays.The potential of such an interpretation is that the diagrams pro…
ClusteringCharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters
Due to the compelling improvements brought by BERT, many recent representation models adopted the Transformer architecture as their main building block, consequently inheriting the wordpiece tokenization system despite i…
Clinical Concept ExtractionDrug–drug Interaction ExtractionNatural Language InferenceRelation Extraction+1Recognizing Characters in Art History Using Deep Learning
In the field of Art History, images of artworks and their contexts are core to understanding the underlying semantic information. However, the highly complex and sophisticated representation of these artworks makes it di…
Deep LearningDeep Neural Networks - A Brief History
Introduction to deep neural networks and their history.
Extracting Networks of Characters and Places from Written Works with CHAPLIN
We are proposing a tool able to gather information on social networks from narrative texts. Its name is CHAPLIN, CHAracters and PLaces Interaction Network, implemented in VB.NET. Characters and places of the narrative wo…