PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese
Large Language Models (LLMs) exhibit significant variations in performance across linguistic and cultural contexts, underscoring the need for systematic evaluation in diverse languages. In this work, we present the most extensive evaluation of LLMs for the Portuguese language to date. Leveraging our newly introduced PoETa v2 benchmark -- a comprehensive suite of over 40 tasks in Portuguese -- we assess more than 20 models covering a broad spectrum of training scales and computational resources. Our study reveals how computational investment and language-specific adaptation impact performance in Portuguese, while also analyzing performance gaps in comparison to equivalent tasks in English. Through this benchmark and analysis, PoETa v2 lays the groundwork for future research on Portuguese language modeling and evaluation. The benchmark is available at https://github.com/PoETaV2/PoETaV2.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Sabiá: Portuguese Large Language Models
As the capabilities of language models continue to advance, it is conceivable that "one-size-fits-all" model will remain as the main paradigm. For instance, given the vast number of languages worldwide, many of which are…
Cultural Vocal Bursts Intensity PredictionSynthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining
Synthetic data generation through document rewriting has emerged as a promising technique for improving language model pretraining, yet most studies focus on English and do not systematically control for the quality of t…
Synthetic Data GenerationO Poeta Artificial 2.0: Increasing Meaningfulness in a Poetry Generation Twitter bot
From Brazilian Portuguese to European Portuguese
Brazilian Portuguese and European Portuguese are two varieties of the same language and, despite their close similarities, they exhibit several differences. However, there is a significant disproportion in the availabili…
SentenceTranslationCLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility
This paper reports on the development of a leaderboard of Open Large Language Models (LLM) for European Portuguese (PT-PT), and on its associated benchmarks. This leaderboard comes as a way to address a gap in the evalua…