Assessing the Impact of Sequence Length Learning on Classification Tasks for Transformer Encoder Models

2022-12-16 · Jean-Thomas Baillargeon, Luc Lamontagne

Classification algorithms using Transformer architectures can be affected by the sequence length learning problem whenever observations from different classes have a different length distribution. This problem causes models to use sequence length as a predictive feature instead of relying on important textual information. Although most public datasets are not affected by this problem, privately owned corpora for fields such as medicine and insurance may carry this data bias. The exploitation of this sequence length feature poses challenges throughout the value chain as these machine learning models can be used in critical applications. In this paper, we empirically expose this problem and present approaches to minimize its impacts.

📄 PDF Abstract BibTeX arXiv:2212.08399

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Residual Connection 설명 없음

Detecting Neurodegenerative Diseases using Frame-Level Handwriting Embeddings

2025-02-10 · Sarah Laouedj, Yuzhe Wang, Jesus Villalba, Thomas Thebaud 외

In this study, we explored the use of spectrograms to represent handwriting signals for assessing neurodegenerative diseases, including 42 healthy controls (CTL), 35 subjects with Parkinson's Disease (PD), 21 with Alzhei…

Binary Classification Classification

What is the Best Sequence Length for BABYLM?

2025-10-22 · Suchir Salhan, Richard Diehl Martinez, Zébulon Goriely, Paula Buttery arxiv

Transformer language models typically operate with a fixed-length context window, which has grown in step with large-scale pretraining datasets. In the BabyLM Challenge, however, many past submissions have defaulted to u…

Does Dialog Length matter for Next Response Selection task? An Empirical Study

2021-01-24 · Jatin Ganhotra, Sachindra Joshi

In the last few years, the release of BERT, a multilingual transformer based model, has taken the NLP community by storm. BERT-based models have achieved state-of-the-art results on various NLP tasks, including dialog ta…

Video Quality Assessment for Computer Graphics Applications

2010-12-01 · ACM Transactions on Graphics 2010 12 · Tunc Ozan Aydin, Martin Cadik, Karol Myszkowski, Hans-Peter Seidel

Numerous current Computer Graphics methods produce video sequences as their outcome. The merit of these methods is often judged by assessing the quality of a set of results through lengthy user studies. We present a full…

Tone Mapping Video Compression Video Quality Assessment

Unsupervised Representation Learning of DNA Sequences

2019-06-07 · Vishal Agarwal, N Jayanth Kumar Reddy, Ashish Anand

Recently several deep learning models have been used for DNA sequence based classification tasks. Often such tasks require long and variable length DNA sequences in the input. In this work, we use a sequence-to-sequence …

Classification General Classification Representation Learning