paper-with-me

Papers

Chain-of-Thought Enhanced Shallow Transformers for Wireless Symbol Detection

2025-06-26 · Li Fan, Peng Wang, Jing Yang, Cong Shen

Transformers have shown potential in solving wireless communication problems, particularly via in-context learning (ICL), where models adapt to new tasks through prompts without requiring model updates. However, prior ICL-based Transformer models rely on deep architectures with many layers to achieve satisfactory performance, resulting in substantial storage and computational costs. In this work, we propose CHain Of thOught Symbol dEtection (CHOOSE), a CoT-enhanced shallow Transformer framework for wireless symbol detection. By introducing autoregressive latent reasoning steps within the hidden space, CHOOSE significantly improves the reasoning capacity of shallow models (1-2 layers) without increasing model depth. This design enables lightweight Transformers to achieve detection performance comparable to much deeper models, making them well-suited for deployment on resource-constrained mobile devices. Experimental results demonstrate that our approach outperforms conventional shallow Transformers and achieves performance comparable to that of deep Transformers, while maintaining storage and computational efficiency. This represents a promising direction for implementing Transformer-based algorithms in wireless receivers with limited computational resources.

📄 PDF Abstract BibTeX arXiv:2506.21093

Code (0)

등록된 구현이 없습니다.

Tasks

Computational EfficiencyIn-Context Learning

Methods 이 논문이 사용한 방법론

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…
Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…
Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…
Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…
BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…
Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…
Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…
Transformer A Transformer is a model architecture that eschews recurrence and instead relies entirely on an [attention…

Similar Papers 제목 키워드 기반

Multi-head Transformers Provably Learn Symbolic Multi-step Reasoning via Gradient Descent

2025-08-11 · Tong Yang, Yu Huang, Yingbin Liang, Yuejie Chi arxiv

Transformers have demonstrated remarkable capabilities in multi-step reasoning tasks. However, understandings of the underlying mechanisms by which they acquire these abilities through training remain limited, particular…

Deductive Logic in Language Models: Horizontal vs Vertical Reasoning

2025-10-10 · Davide Maltoni, Matteo Ferrara arxiv

Recent language models exhibit significant logical reasoning abilities, yet the mechanisms supporting deductive inference remain poorly understood. This paper studies small transformer-based language models trained from …

Logical ReasoningDecision Making

CoT-ICL Lab: A Petri Dish for Studying Chain-of-Thought Learning from In-Context Demonstrations

2025-02-21 · Vignesh Kothapalli, Hamed Firooz, Maziar Sanjabi

We introduce CoT-ICL Lab, a framework and methodology to generate synthetic tokenized datasets and systematically study chain-of-thought (CoT) in-context learning (ICL) in language models. CoT-ICL Lab allows fine grained…

DecoderDiversityIn-Context Learning

Theoretical limitations of multi-layer Transformer

2024-12-04 · Lijie Chen, Binghui Peng, Hongxun Wu

Transformers, especially the decoder-only variants, are the backbone of most modern large language models; yet we do not have much understanding of their expressive power except for the simple $1$-layer case. Due to the …

Decoder

Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers

2025-02-04 · Alireza Amiri, Xinting Huang, Mark Rofin, Michael Hahn

Chain-of-thought reasoning and scratchpads have emerged as critical tools for enhancing the computational capabilities of transformers. While theoretical results show that polynomial-length scratchpads can extend transfo…

Hard Attention