Semantics Alignment via Split Learning for Resilient Multi-User Semantic Communication

2023-10-13 · Jinhyuk Choi, Jihong Park, Seung-Woo Ko, Jinho Choi, Mehdi Bennis, Seong-Lyun Kim

Recent studies on semantic communication commonly rely on neural network (NN) based transceivers such as deep joint source and channel coding (DeepJSCC). Unlike traditional transceivers, these neural transceivers are trainable using actual source data and channels, enabling them to extract and communicate semantics. On the flip side, each neural transceiver is inherently biased towards specific source data and channels, making different transceivers difficult to understand intended semantics, particularly upon their initial encounter. To align semantics over multiple neural transceivers, we propose a distributed learning based solution, which leverages split learning (SL) and partial NN fine-tuning techniques. In this method, referred to as SL with layer freezing (SLF), each encoder downloads a misaligned decoder, and locally fine-tunes a fraction of these encoder-decoder NN layers. By adjusting this fraction, SLF controls computing and communication costs. Simulation results confirm the effectiveness of SLF in aligning semantics under different source data and channel dissimilarities, in terms of classification accuracy, reconstruction errors, and recovery time for comprehending intended semantics from misalignment.

📄 PDF Abstract BibTeX arXiv:2310.09394

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder Semantic Communication

Methods 이 논문이 사용한 방법론

FLIP https://developer.nvidia.com/blog/flip-a-difference-evaluator-for-alternating-images/

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Token Communication-Driven Multimodal Large Models in Resource-Constrained Multiuser Networks

2025-05-06 · Junhe Zhang, Wanli Ni, Pengwei Wang, Dongyu Wang

The proliferation of intelligent applications at the wireless edge, alongside the exponential growth of multimodal data, poses challenges for deploying multimodal large models (MLMs) in resource-constrained networks. The…

SSPA: Split-and-Synthesize Prompting with Gated Alignments for Multi-Label Image Recognition

2024-07-30 · Hao Tan, Zichang Tan, Jun Li, Jun Wan 외

Multi-label image recognition is a fundamental task in computer vision. Recently, Vision-Language Models (VLMs) have made notable advancements in this area. However, previous methods fail to effectively leverage the rich…

In-Context Learning Multi-Label Image Recognition

Robustness of Vision Language Models Against Split-Image Harmful Input Attacks

2026-02-08 · Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu, Shagufta Mehnaz arxiv

Vision-Language Models (VLMs) are now a core part of modern AI. Recent work proposed several visual jailbreak attacks using single/ holistic images. However, contemporary VLMs demonstrate strong robustness against such a…

Knowledge Distillation

AliMark: Enhancing Robustness of Sentence-Level Watermarking Against Text Paraphrasing

2026-05-28 · Yuexin Li, Wenjie Qu, Linyu Wu, Yulin Chen 외 arxiv

Existing sentence-level watermarking methods enhance robustness to paraphrasing by anchoring watermarks in sentence semantics. However, their prefix-based designs remain vulnerable to structural perturbations, such as se…

NNSplitter: An Active Defense Solution for DNN Model via Automated Weight Obfuscation

2023-04-28 · Tong Zhou, Yukui Luo, Shaolei Ren, Xiaolin Xu

As a type of valuable intellectual property (IP), deep neural network (DNN) models have been protected by techniques like watermarking. However, such passive model protection cannot fully prevent model abuse. In this wor…