CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility
This paper reports on the development of a leaderboard of Open Large Language Models (LLM) for European Portuguese (PT-PT), and on its associated benchmarks. This leaderboard comes as a way to address a gap in the evaluation of LLM for European Portuguese, which so far had no leaderboard dedicated to this variant of the language. The paper also reports on novel benchmarks, including some that address aspects of performance that so far have not been available in benchmarks for European Portuguese, namely model safeguards and alignment to Portuguese culture. The leaderboard is available at https://huggingface.co/spaces/PORTULAN/portuguese-llm-leaderboard.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Infrastructure for the Science and Technology of Language PORTULAN CLARIN
This paper presents the PORTULAN CLARIN Research Infrastructure for the Science and Technology of Language, which is part of the European research infrastructure CLARIN ERIC as its Portuguese national node, and belongs t…
Universal Grammatical Dependencies for Portuguese with CINTIL Data, LX Processing and CLARIN support
The grammatical framework for the mapping between linguistic form and meaning representation known as Universal Dependencies relies on a non-constituency syntactic analysis that is centered on the notion of grammatical r…
MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese
Text embeddings for Portuguese have no dedicated benchmark: evaluation rests on translated corpora such as English MS MARCO or on thin multilingual coverage, with native tasks scattered and unconsolidated. We introduce M…
Semantic Textual SimilarityGlórIA - A Generative and Open Large Language Model for Portuguese
Significant strides have been made in natural language tasks, largely attributed to the emergence of powerful large language models (LLMs). These models, pre-trained on extensive and diverse corpora, have become increasi…
DecoderLanguage ModelingLanguage ModellingLarge Language ModelNumber game
CLARIN (Common Language Resources and Technology Infrastructure) is regarded as one of the most important European research infrastructures, offering and promoting a wide array of useful services for (digital) research i…
Marketing