OSCAR
홈페이지 · 논문 64편
OSCAR or Open Super-large Crawled ALMAnaCH coRpus is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture. The dataset used for training multilingual models such as BART incorporates 138 GB of text.
Texts EnglishFrenchSpanishGermanItalianChineseBengaliMultilingualJapaneseRussianPortugueseAfrikaansAlbanianAmharicArabicArmenianBasqueBelarusianBretonBulgarianCatalanCroatianCzechDanishDutchErzyaEstonianFinnishGalicianHebrewHindiHungarianIcelandicIndonesianIrishKazakhKoreanLatinLatvianLithuanianMalteseMarathiModern GreekNorwegianPersianPolishRomanianRussia BuriatSanskritScottish GaelicSerbianSlovakSlovenianSwedishTagalogTamilTeluguThaiTurkishUighurUkrainianUpper SorbianUrduVietnameseWelshYorubaYue ChineseSwahiliAragoneseEgyptian ArabicAssameseAsturianAvaricSouth AzerbaijaniAzerbaijaniBashkirBavarianCentral BikolTibetanBosnianBishnupriyaChavacanoCebuanoChechenChuvashCentral KurdishCornishDimli (individual language)DhivehiLower SorbianEsperantoNorthern FrisianWestern FrisianGoan KonkaniGuaraniGujaratiHaitianSerbo-CroatianIdoInterlingueIlokoJavaneseLojbanKannadaGeorgianCentral KhmerKirghizKomiKarachay-BalkarKurdishLaoLezghianLimburganLombardNorthern LuriLuxembourgishMaithiliMalayalamEastern MariMinangkabauMacedonianMalagasyMongolianWestern MariMirandeseBurmeseMazanderaniNeapolitanLow GermanNewariNorwegian NynorskOccitan (post 1500)Oriya (macrolanguage)OssetianPampangaPunjabiPiemonteseWestern PanjabiPushtoQuechuaRomanshYakutSicilianSinhalaSindhiSomaliSundaneseTatarTajikTurkmenTuvinianUzbekVenetianVolapükWaray (Philippines)WalloonWu ChineseKalmykMingrelianYiddishMalay (individual language)