NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scripts
Indonesia is rich in languages and scripts. However, most NLP progress has been made using romanized text. In this paper, we present NusaAksara, a novel public benchmark for Indonesian languages that includes their original scripts. Our benchmark covers both text and image modalities and encompasses diverse tasks such as image segmentation, OCR, transliteration, translation, and language identification. Our data is constructed by human experts through rigorous steps. NusaAksara covers 8 scripts across 7 languages, including low-resource languages not commonly seen in NLP benchmarks. Although unsupported by Unicode, the Lampung script is included in this dataset. We benchmark our data across several models, from LLMs and VLMs such as GPT-4o, Llama 3.2, and Aya 23 to task-specific systems such as PP-OCR and LangID, and show that most NLP technologies cannot handle Indonesia's local scripts, with many achieving near-zero performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Image SegmentationLanguage IdentificationOptical Character Recognition (OCR)Semantic SegmentationTransliterationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Does Language Shift Break Medical Vision-Language Models? Indonesian Radiology Visual Question Answering Case Study
Medical Vision-Language Models (VLMs) are typically evaluated on English radiology visual question answering benchmarks, leaving their robustness under non-English clinical language largely unexplored. We introduce IndoR…
Visual Question AnsweringMERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking
This paper introduces MERLIN, a novel testbed system for the task of Multilingual Multimodal Entity Linking. The created dataset includes BBC news article titles, paired with corresponding images, in five languages: Hind…
Entity LinkingImproving Indonesian Text Classification Using Multilingual Language Model
Compared to English, the amount of labeled data for Indonesian text classification tasks is very small. Recently developed multilingual language models have shown its ability to create multilingual representations effect…
ClassificationGeneral ClassificationLanguage ModelingLanguage Modelling+4MSVD-Indonesian: A Benchmark for Multimodal Video-Text Tasks in Indonesian
Multimodal learning on video and text data has been receiving growing attention from many researchers in various research tasks, including text-to-video retrieval, video-to-text retrieval, and video captioning. Although …
Cross-Lingual TransferRetrievalText RetrievalText to Video Retrieval+5LoraxBench: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages
As one of the world's most populous countries, with 700 languages spoken, Indonesia is behind in terms of NLP progress. We introduce LoraxBench, a benchmark that focuses on low-resource languages of Indonesia and covers …
Reading Comprehension