Continual Pretraining
3개 벤치마크 · 논문 147편 · 이 태스크의 논문 보기 →
Benchmarks
Most implemented
Rho-1: Not All Tokens Are What You Need
Continual Training of Language Models for Few-Shot Learning
Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts
Effective Long-Context Scaling of Foundation Models
Towards Geospatial Foundation Models via Continual Pretraining
Continual Pre-training of Language Models
Papers
Studying Image Tokenizers as Visual Languages in Unified Multimodal Models
Image tokenizers define the ``visual language'' of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how vi…
Continual PretrainingMetaboLLM: a metabolomics-specialized large language model for biochemical knowledge integration and predictive metabolite graph construction
Metabolomics knowledge is distributed across heterogeneous resources and remains difficult to translate into predictive representations. We developed MetaboLLM, a metabolomics-specialized large language model adapted thr…
Continual PretrainingAXIS: A Growable Community-Driven Data Engine for Scalable Robot Manipulation
Learning effective robot manipulation policies requires diverse, high-quality demonstrations, yet existing data pipelines are often difficult to scale because they rely on specialized hardware, centralized operators, or …
Continual PretrainingRobot ManipulationThe Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability
Fine-tuning has been widely used to adapt large language models (LLMs) for domain-specific tasks. Parameter efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) are frequently used to reduce computatio…
Continual PretrainingDiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation
Large language models increasingly \emph{understand} dialectal English, yet still \emph{produce} only standard, US-leaning English, leaving dialectal generation, the harder half of the problem, largely unaddressed. We in…
Continual PretrainingLACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning
LLMs memorize sensitive training data, including personally identifiable information (PII), creating a pressing need for reliable post hoc removal methods. Unlearning has emerged as a promising solution, with state-of-th…
Continual Pretraining