Vision Transformer-based Semantic Communications With Importance-Aware Quantization

2024-12-08 · Joohyuk Park, Yongjeong Oh, Yongjune Kim, Yo-Seb Jeon

Semantic communications provide significant performance gains over traditional communications by transmitting task-relevant semantic features through wireless channels. However, most existing studies rely on end-to-end (E2E) training of neural-type encoders and decoders to ensure effective transmission of these semantic features. To enable semantic communications without relying on E2E training, this paper presents a vision transformer (ViT)-based semantic communication system with importance-aware quantization (IAQ) for wireless image transmission. The core idea of the presented system is to leverage the attention scores of a pretrained ViT model to quantify the importance levels of image patches. Based on this idea, our IAQ framework assigns different quantization bits to image patches based on their importance levels. This is achieved by formulating a weighted quantization error minimization problem, where the weight is set to be an increasing function of the attention score. Then, an optimal incremental allocation method and a low-complexity water-filling method are devised to solve the formulated problem. Our framework is further extended for realistic digital communication systems by modifying the bit allocation problem and the corresponding allocation methods based on an equivalent binary symmetric channel (BSC) model. Simulations on single-view and multi-view image classification tasks show that our IAQ framework outperforms conventional image compression methods in both error-free and realistic communication scenarios.

📄 PDF Abstract BibTeX arXiv:2412.06038

Code (0)

등록된 구현이 없습니다.

Tasks

image-classification Image Classification Image Compression Quantization Semantic Communication

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Residual Connection 설명 없음

Semantic Importance-Aware Communications Using Pre-trained Language Models

2023-02-12 · Shuaishuai Guo, Yanhu Wang, Shujing Li, Nasir Saeed

This letter proposes a semantic importance-aware communication (SIAC) scheme using pre-trained language models (e.g., ChatGPT, BERT, etc.). Specifically, we propose a cross-layer design with a pre-trained language model …

Language Modeling Language Modelling Semantic Communication

Semantic Importance-Aware Communications with Semantic Correction Using Large Language Models

2024-05-25 · Shuaishuai Guo, Yanhu Wang, Jia Ye, Anbang Zhang 외

Semantic communications, a promising approach for agent-human and agent-agent interactions, typically operate at a feature level, lacking true semantic understanding. This paper explores understanding-level semantic comm…

Language Modeling Language Modelling Large Language Model

Will 6G be Semantic Communications? Opportunities and Challenges from Task Oriented and Secure Communications to Integrated Sensing

2024-01-03 · Yalin E. Sagduyu, Tugba Erpek, Aylin Yener, Sennur Ulukus

This paper explores opportunities and challenges of task (goal)-oriented and semantic communications for next-generation (NextG) communication networks through the integration of multi-task learning. This approach employ…

Federated Learning Multi-Task Learning Robust Design

Demo: Real-Time Semantic Communications with a Vision Transformer

2022-05-08 · Hanju Yoo, Taehun Jung, Linglong Dai, Songkuk Kim 외

Semantic communications are expected to enable the more effective delivery of meaning rather than a precise transfer of symbols. In this paper, we propose an end-to-end deep neural network-based architecture for image tr…

Multimodal LLM Integrated Semantic Communications for 6G Immersive Experiences

2025-07-07 · Yusong Zhang, Yuxuan Sun, Lei Guo, Wei Chen 외 arxiv

6G networks promise revolutionary immersive communication experiences including augmented reality (AR), virtual reality (VR), and holographic communications. These applications demand high-dimensional multimodal data tra…

Visual Question Answering Image Generation