A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT
Diversity has been gaining interest in the NLP community in recent years. At the same time, state-of-the-art transformer models such as ModernBERT use very large pre-training datasets, which are driven by size rather than by diversity. This summons for an investigation of the impact of diversity on the ModernBERT pre-training. We do so in this study, with the express intent of reducing pre-training dataset size, while retaining at least comparable performance. We compare diversity-driven sampling algorithms, so as to pick the best one. We find that diversity-driven sampling allows in some tasks to gain 10 points relative to randomly-sampled pre-training data of commensurate size. We also see that a model pre-trained for 483h on a diversity-driven dataset of 150M tokens can yield a commensurate performance to a model pre-trained for 1,775h on a randomly-driven dataset of 2.4B tokens.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Extracting Food Substitutes From Food Diary via Distributional Similarity
In this paper, we explore the problem of identifying substitute relationship between food pairs from real-world food consumption data as the first step towards the healthier food recommendation. Our method is inspired by…
Food recommendationImage-Based Dietary Assessment: A Healthy Eating Plate Estimation System
The nutritional quality of diets has significantly deteriorated over the past two to three decades, a decline often underestimated by the people. This deterioration, coupled with a hectic lifestyle, has contributed to es…
Image SegmentationSemantic SegmentationEnvironmental impacts, nutritional profiles, and retail prices of commonly sold retail food items in 181 countries: an observational study
Affordability is often seen as a barrier to consuming sustainable diets. This study provides the first worldwide test of how retail food prices relate to empirically estimated environmental impacts and nutritional profil…
AttributeClimate impacts and monetary costs of healthy diets worldwide
About 2.8 billion people worldwide cannot afford the least expensive foods required for a healthy diet. Since 2020, the Cost and Affordability of a Healthy Diet (CoAHD) has been published for all countries by FAO and the…
NutritionInvestigating Health-Aware Smart-Nudging with Machine Learning to Help People Pursue Healthier Eating-Habits
Food-choices and eating-habits directly contribute to our long-term health. This makes the food recommender system a potential tool to address the global crisis of obesity and malnutrition. Over the past decade, artifici…
Decision MakingRecommendation Systems