Vision Transformers for Small Histological Datasets Learned through Knowledge Distillation

2023-05-27 · Neel Kanwal, Trygve Eftestol, Farbod Khoraminia, Tahlita CM Zuiverloon, Kjersti Engan

Computational Pathology (CPATH) systems have the potential to automate diagnostic tasks. However, the artifacts on the digitized histological glass slides, known as Whole Slide Images (WSIs), may hamper the overall performance of CPATH systems. Deep Learning (DL) models such as Vision Transformers (ViTs) may detect and exclude artifacts before running the diagnostic algorithm. A simple way to develop robust and generalized ViTs is to train them on massive datasets. Unfortunately, acquiring large medical datasets is expensive and inconvenient, prompting the need for a generalized artifact detection method for WSIs. In this paper, we present a student-teacher recipe to improve the classification performance of ViT for the air bubbles detection task. ViT, trained under the student-teacher framework, boosts its performance by distilling existing knowledge from the high-capacity teacher model. Our best-performing ViT yields 0.961 and 0.911 F1-score and MCC, respectively, observing a 7% gain in MCC against stand-alone training. The proposed method presents a new perspective of leveraging knowledge distillation over transfer learning to encourage the use of customized transformers for efficient preprocessing pipelines in the CPATH systems.

📄 PDF Abstract BibTeX arXiv:2305.17370

Code (1)

NeelKanwal/Vision-Transformers-for-Small-Histological-Datasets-Learned-Through-Knowledge-Distillation 공식 구현 pytorch

Tasks

Airbubbles DetectionAnomaly Detection Artifact DetectionDiagnosticKnowledge Distillation Transfer Learningwhole slide images

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

IBiT: Utilizing Inductive Biases to Create a More Data Efficient Attention Mechanism

2025-09-24 · Adithya Giri arxiv

In recent years, Transformer-based architectures have become the dominant method for Computer Vision applications. While Transformers are explainable and scale well with dataset size, they lack the inductive biases of Co…

Knowledge Distillation

How to Train Vision Transformer on Small-scale Datasets?

2022-10-13 · Hanan Gani, Muzammal Naseer, Mohammad Yaqub

Vision Transformer (ViT), a radically different architecture than convolutional neural networks offers multiple advantages including design simplicity, robustness and state-of-the-art performance on many vision tasks. Ho…

Scaling Federated Learning Solutions with Kubernetes for Synthesizing Histopathology Images

2025-04-05 · Andrei-Alexandru Preda, Iulian-Marius Tăiatu, Dumitru-Clementin Cercel

In the field of deep learning, large architectures often obtain the best performance for many tasks, but also require massive datasets. In the histological domain, tissue images are expensive to obtain and constitute sen…

Federated Learning image-classification Image Classification

Training Vision Transformers with Only 2040 Images

2022-01-26 · Yun-Hao Cao, Hao Yu, Jianxin Wu

Vision Transformers (ViTs) is emerging as an alternative to convolutional neural networks (CNNs) for visual recognition. They achieve competitive results with CNNs but the lack of the typical convolutional inductive bias…

Inductive Bias

Convolutional Initialization for Data-Efficient Vision Transformers

2024-01-23 · Jianqiao Zheng, Xueqian Li, Simon Lucey

Training vision transformer networks on small datasets poses challenges. In contrast, convolutional neural networks (CNNs) can achieve state-of-the-art performance by leveraging their architectural inductive bias. In thi…