홈 › Papers

MILO: Model-Agnostic Subset Selection Framework for Efficient Model Training and Tuning

2023-01-30 · KrishnaTeja Killamsetty, Alexandre V. Evfimievski, Tejaswini Pedapati, Kiran Kate, Lucian Popa, Rishabh Iyer

Training deep networks and tuning hyperparameters on large datasets is computationally intensive. One of the primary research directions for efficient training is to reduce training costs by selecting well-generalizable subsets of training data. Compared to simple adaptive random subset selection baselines, existing intelligent subset selection approaches are not competitive due to the time-consuming subset selection step, which involves computing model-dependent gradients and feature embeddings and applies greedy maximization of submodular objectives. Our key insight is that removing the reliance on downstream model parameters enables subset selection as a pre-processing step and enables one to train multiple models at no additional cost. In this work, we propose MILO, a model-agnostic subset selection framework that decouples the subset selection from model training while enabling superior model convergence and performance by using an easy-to-hard curriculum. Our empirical results indicate that MILO can train models $3\times - 10 \times$ faster and tune hyperparameters $20\times - 75 \times$ faster than full-dataset training or tuning without compromising performance.

📄 PDF Abstract BibTeX arXiv:2301.13287

Code (0)

등록된 구현이 없습니다.

Tasks

model

Feature subset selection for kernel SVM classification via mixed-integer optimization

2022-05-28 · Ryuta Tamura, Yuichi Takano, Ryuhei Miyashiro

We study the mixed-integer optimization (MIO) approach to feature subset selection in nonlinear kernel support vector machines (SVMs) for binary classification. First proposed for linear regression in the 1970s, this app…

Binary Classification Computational Efficiency

The Discrete Dantzig Selector: Estimating Sparse Linear Models via Mixed Integer Linear Optimization

2015-08-08 · Rahul Mazumder, Peter Radchenko

We propose a novel high-dimensional linear regression estimator: the Discrete Dantzig Selector, which minimizes the number of nonzero regression coefficients subject to a budget on the maximal absolute correlation betwee…

regression Variable Selection

MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators

2025-04-03 · Beichen Huang, Yueming Yuan, Zelei Shao, Minjia Zhang

A critical approach for efficiently deploying Mixture-of-Experts (MoE) models with massive parameters is quantization. However, state-of-the-art MoE models suffer from non-negligible accuracy loss with extreme quantizati…

Mixture-of-Experts Quantization

Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs

2024-09-16 · Yifan Wang, David Stevens, Pranay Shah, WenWen Jiang 외

The growing demand for AI training data has transformed data annotation into a global industry, but traditional approaches relying on human annotators are often time-consuming, labor-intensive, and prone to inconsistent …

MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgetting

2026-01-28 · Jing Xu, Minglin Wu, Xueyuan Chen, Xixin Wu 외 arxiv

Self-supervised learning (SSL) has greatly advanced speech representation learning, but multilingual SSL models remain constrained to languages encountered during pretraining. Retraining from scratch to incorporate new l…

Self-Supervised Learning Representation Learning