Parameter-Efficient Quantized Mixture-of-Experts Meets Vision-Language Instruction Tuning for Semiconductor Electron Micrograph Analysis

2024-08-27 · Sakhinana Sagar Srinivas, Chidaksh Ravuru, Geethan Sannidhi, Venkataramana Runkana

Semiconductors, crucial to modern electronics, are generally under-researched in foundational models. It highlights the need for research to enhance the semiconductor device technology portfolio and aid in high-end device fabrication. In this paper, we introduce sLAVA, a small-scale vision-language assistant tailored for semiconductor manufacturing, with a focus on electron microscopy image analysis. It addresses challenges of data scarcity and acquiring high-quality, expert-annotated data. We employ a teacher-student paradigm, using a foundational vision language model like GPT-4 as a teacher to create instruction-following multimodal data for customizing the student model, sLAVA, for electron microscopic image analysis tasks on consumer hardware with limited budgets. Our approach allows enterprises to further fine-tune the proposed framework with their proprietary data securely within their own infrastructure, protecting intellectual property. Rigorous experiments validate that our framework surpasses traditional methods, handles data shifts, and enables high-throughput screening.

📄 PDF Abstract BibTeX arXiv:2408.15305

Code (0)

등록된 구현이 없습니다.

Tasks

Instruction FollowingLanguage ModelingLanguage ModellingMixture-of-Experts

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Adam 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Attention 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Residual Connection 설명 없음

Multi-Head Attention 설명 없음

YOLO Meets Mixture-of-Experts: Adaptive Expert Routing for Robust Object Detection

2025-11-17 · Ori Meiraz, Sharon Shalev, Avishai Weizman arxiv

This paper presents a novel Mixture-of-Experts framework for object detection, incorporating adaptive routing among multiple YOLOv9-T experts to enable dynamic feature specialization and achieve higher mean Average Preci…

Robust Object Detection

Mixture of Experts Meets Prompt-Based Continual Learning

2024-05-23 · Minh Le, An Nguyen, Huy Nguyen, Trang Nguyen 외

Exploiting the power of pre-trained models, prompt-based approaches stand out compared to other continual learning solutions in effectively preventing catastrophic forgetting, even with very few learnable parameters and …

Continual Learning Mixture-of-Experts

Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

2026-04-14 · NVIDIA, :, Aakshita Chandiramani, Aaron Blakeman 외 arxiv

We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemotron 3 Super is the first model in the N…

Reinforcement Learning

On the effectiveness of discrete representations in sparse mixture of experts

2024-11-28 · Giang Do, Kha Pham, Hung Le, Truyen Tran

Sparse mixture of experts (SMoE) is an effective solution for scaling up model capacity without increasing the computational costs. A crucial component of SMoE is the router, responsible for directing the input to releva…

Mixture-of-Experts Quantization

ButterflyViT: 354$\times$ Expert Compression for Edge Vision Transformers

2026-03-06 · Aryan Karmore arxiv

Deploying sparse Mixture of Experts(MoE) Vision Transformers remains a challenge due to linear expert memory scaling. Linear memory scaling stores $N$ independent expert weight matrices requiring $\mathcal{O}(N_E \cdot d…

Image Classification