Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression
Teacher-Student Knowledge Transfer (KT) is ubiquitous in modern machine learning, ranging from classical model compression via Knowledge Distillation (KD) to the emergent phenomenon of Weak-to-Strong (W2S) generalization…
Knowledge DistillationModel CompressionMulti-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors
Knowledge distillation is a powerful method for model compression, enabling the efficient deployment of complex deep learning models (teachers), including large language models. However, its underlying statistical mechan…
Knowledge DistillationImage ClassificationBayesian InferenceModel CompressionStreamSplit: Continuous Audio Representation Learning via Uncertainty-Guided Adaptive Splitting
Large-batch Contrastive Learning (CL), the foundation of modern representation learning, is fundamentally incompatible with the volatile resource constraints of edge devices. This conflict creates a dilemma: small on-dev…
Representation LearningReinforcement LearningContrastive LearningModel CompressionMTL-FNO: A Lightweight Multi-Task Fourier Neural Operator for Sparse Field Reconstruction
Efficient onboard multi-field sparse reconstruction is essential for the autonomous operation of aerospace vehicles. While existing deep learning models exhibit promise for single-field reconstruction, deploying multiple…
Model CompressionThe Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works
Knowledge distillation (KD) transfers knowledge from a large teacher model to a smaller student. In language modeling, the student is trained either on tokens sampled from the teacher (hard labels) or the teacher's full …
Knowledge DistillationModel CompressionImplicitTerrainV2: Wavelet-Guided Spatially Adaptive Neural Terrain Representation
Digital elevation models (DEMs) underpin terrain analysis in Geographic Information Systems (GIS), but in their common raster form, they rely on interpolation for off-grid sampling and finite-difference operators for der…
Model CompressionMGVQ: Synergizing Multi-dimensional Sensitivity-Aware and Gradient-Hessian Fusion for Vector Quantization
Vision-Language Models (VLMs) achieve outstanding performance, yet their huge model size severely hinders deployment on edge devices with limited resources. As an efficient model compression technique, vector quantizatio…
Model CompressionFTerViT: Fully Ternary Vision Transformer
Ternary Vision Transformers offer substantial model compression, however state-of-the-art methods only ternarize the encoder layers, leaving patch embeddings, LayerNorm parameters, and classifier heads in full precision.…
Knowledge DistillationModel CompressionAutoMCU: Feasibility-First MCU Neural Network Customization via LLM-based Multi-Agent Systems
Deploying neural networks on microcontroller units (MCUs) is critical for edge intelligence but remains challenging due to tight memory, storage, and computation constraints. Existing approaches, such as model compressio…
Neural Architecture SearchModel CompressionK-Quantization and its Impact on Output Performance
Recent advancements in large language models (LLMs) have shown their remarkable capacities in many NLP tasks. However, their substantial size often presents challenges for deployment. This necessitates efficient techniqu…
Reading ComprehensionModel CompressionSAFE-SVD: Sensitivity-Aware Fidelity-Enforcing SVD for Physics Foundation Models
We propose a new method for compressing physics foundation models (PFMs) which is a new trend in AI for Science. While model compression is essential for reducing memory use and accelerating inference in large foundation…
Model CompressionCAR-SAM: Cross-Attention Reconstruction for Post-Training Quantization of the Segment Anything Model
Segment Anything Models (SAMs) are extensively used in computer vision for universal image segmentation, but deploying them on resource-constrained devices is challenging due to their high computational and memory demand…
Image SegmentationModel CompressionCharacterizing Learning in Deep Neural Networks using Tractable Algorithmic Complexity Analysis
Training large-scale deep neural networks (DNNs) is resource-intensive, making model compression a practical necessity. The widely accepted ''learning as compression'' hypothesis posits that training induces structure in…
Model CompressionNonlinear Bipolar Compensation: Handling Outliers in Post-Training Quantization
Network quantization has emerged as one of the most practical model compression techniques, which significantly reduces a model's memory and compute consumption by mapping floating-point numbers to low-bit representation…
Model CompressionEfficient LLM-based Advertising via Model Compression and Parallel Verification
Large language models (LLMs) have shown remarkable potential in advertising scenarios such as ad creative generation and targeted advertising. However, deploying LLMs in real-time advertising systems poses significant ch…
Model CompressionRobust Basis Spline Decoupling for the Compression of Transformer Models
Decoupling is a powerful modeling paradigm for representing multivariate functions as compositions of linear transformations and univariate nonlinear functions. A single-layer decoupling can be viewed as a fully connecte…
Neural Network CompressionModel CompressionXTinyU-Net: Training-Free U-Net Scaling via Initialization-Time Sensitivity
While U-Net architectures remain the gold standard for medical image segmentation, their deployment in resource-constrained environments demands aggressive model compression. However, finding an optimally efficient confi…
Medical Image SegmentationModel CompressionEdge Deep Learning in Computer Vision and Medical Diagnostics: A Comprehensive Survey
Edge deep learning, a paradigm change reconciling edge computing and deep learning, facilitates real-time decision making attuned to environmental factors through the close integration of computational resources and data…
Model CompressionDecision MakingHCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices
LLMs often struggle with memory-constrained deployment on consumer-grade hardware due to their massive parameter sizes. While existing solutions such as model compression and offloading improve deployment feasibility, th…
Model CompressionUnified Multimodal Visual Tracking with Dual Mixture-of-Experts
Multimodal visual object tracking can be divided into to several kinds of tasks (e.g. RGB and RGB+X tracking), based on the input modality. Existing methods often train separate models for each modality or rely on pretra…
Visual Object TrackingModel CompressionVisual Tracking