CC100

홈페이지 · 논문 115편

This corpus comprises of monolingual data for 100+ languages and also includes data for romanized languages. This was constructed using the urls and paragraph indices provided by the CC-Net repository by processing January-December 2018 Commoncrawl snapshots. Each file comprises of documents separated by double-newlines and paragraphs within the same document separated by a newline. The data is generated using the open source CC-Net repository. Source: [Unsupervised Cross-lingual Representation Learning at Scale](/paper/unsupervised-cross-lingual-representation-1)