LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs

2025-03-04 · Jianghao Chen, Junhong Wu, Yangyifan Xu, Jiajun Zhang

Long-context modeling has drawn more and more attention in the area of Large Language Models (LLMs). Continual training with long-context data becomes the de-facto method to equip LLMs with the ability to process long inputs. However, it still remains an open challenge to measure the quality of long-context training data. To address this issue, we propose a Long-context data selection framework with Attention-based Dependency Measurement (LADM), which can efficiently identify high-quality long-context data from a large-scale, multi-domain pre-training corpus. LADM leverages the retrieval capabilities of the attention mechanism to capture contextual dependencies, ensuring a comprehensive quality measurement of long-context data. Experimental results show that our LADM framework significantly boosts the performance of LLMs on multiple long-context tasks with only 1B tokens for continual training.

📄 PDF Abstract BibTeX arXiv:2503.02502

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

GLADMamba: Unsupervised Graph-Level Anomaly Detection Powered by Selective State Space Model

2025-03-23 · Yali Fu, Jindong Li, Qi Wang, Qianli Xing

Unsupervised graph-level anomaly detection (UGLAD) is a critical and challenging task across various domains, such as social network analysis, anti-cancer drug discovery, and toxic molecule identification. However, exist…

Anomaly Detection Drug Discovery Mamba State Space Models

CLaDMoP: Learning Transferrable Models from Successful Clinical Trials via LLMs

2025-05-24 · Yiqing Zhang, Xiaozhong Liu, Fabricio Murai

Many existing models for clinical trial outcome prediction are optimized using task-specific loss functions on trial phase-specific data. While this scheme may boost prediction for common diseases and drugs, it can hinde…

Large Language Model parameter-efficient fine-tuning Prediction

AA-DLADMM: An Accelerated ADMM-based Framework for Training Deep Neural Networks

2024-01-08 · Zeinab Ebrahimi, Gustavo Batista, Mohammad Deghat

Stochastic gradient descent (SGD) and its many variants are the widespread optimization algorithms for training deep neural networks. However, SGD suffers from inevitable drawbacks, including vanishing gradients, lack of…

Linearized Alternating Direction Method with Parallel Splitting and Adaptive Penalty for Separable Convex Programs in Machine Learning

2013-10-18 · Zhouchen Lin, Risheng Liu, Huan Li

Many problems in machine learning and other fields can be (re)for-mulated as linearly constrained separable convex programs. In most of the cases, there are multiple blocks of variables. However, the traditional alternat…

Distributed Computing parameter estimation

Differentiable Linearized ADMM

2019-05-15 · Xingyu Xie, Jianlong Wu, Zhisheng Zhong, Guangcan Liu 외

Recently, a number of learning-based optimization methods that combine data-driven architectures with the classical optimization algorithms have been proposed and explored, showing superior empirical performance in solvi…