Continual Pretraining
3개 벤치마크 · 논문 149편 · 이 태스크의 논문 보기 →
Benchmarks
Most implemented
Rho-1: Not All Tokens Are What You Need
Continual Training of Language Models for Few-Shot Learning
Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts
Effective Long-Context Scaling of Foundation Models
Towards Geospatial Foundation Models via Continual Pretraining
Continual Pre-training of Language Models
Papers
Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position
The architectural design of Large Language Models (LLMs) is shifting from traditional full-attention-only models to hybrid models, which combine different attention modules to improve long-context efficiency and performa…
Continual PretrainingSpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization
Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing context extension approaches typically apply continued pretraining direc…
Continual PretrainingInformation RetrievalStudying Image Tokenizers as Visual Languages in Unified Multimodal Models
Image tokenizers define the ``visual language'' of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how vi…
Continual PretrainingMetaboLLM: a metabolomics-specialized large language model for biochemical knowledge integration and predictive metabolite graph construction
Metabolomics knowledge is distributed across heterogeneous resources and remains difficult to translate into predictive representations. We developed MetaboLLM, a metabolomics-specialized large language model adapted thr…
Continual PretrainingAXIS: A Growable Community-Driven Data Engine for Scalable Robot Manipulation
Learning effective robot manipulation policies requires diverse, high-quality demonstrations, yet existing data pipelines are often difficult to scale because they rely on specialized hardware, centralized operators, or …
Continual PretrainingRobot ManipulationThe Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability
Fine-tuning has been widely used to adapt large language models (LLMs) for domain-specific tasks. Parameter efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) are frequently used to reduce computatio…
Continual Pretraining