HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarization

2022-03-21 · ACL 2022 5 · Shuyang Cao, Lu Wang

Document structure is critical for efficient information consumption. However, it is challenging to encode it efficiently into the modern Transformer architecture. In this work, we present HIBRIDS, which injects Hierarchical Biases foR Incorporating Document Structure into the calculation of attention scores. We further present a new task, hierarchical question-summary generation, for summarizing salient content in the source document into a hierarchy of questions and summaries, where each follow-up question inquires about the content of its parent question-summary pair. We also annotate a new dataset with 6,153 question-summary hierarchies labeled on long government reports. Experiment results show that our model produces better question-summary hierarchies than comparisons on both hierarchy quality and content coverage, a finding also echoed by human judges. Additionally, our model improves the generation of long-form summaries from lengthy government reports and Wikipedia articles, as measured by ROUGE scores.

📄 PDF Abstract BibTeX arXiv:2203.10741

Code (0)

등록된 구현이 없습니다.

Tasks

Articles Document Summarization

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Residual Connection 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Enhancing Graph Transformers with Hierarchical Distance Structural Encoding

2023-08-22 · Yuankai Luo, Hongkang Li, Lei Shi, Xiao-Ming Wu

Graph transformers need strong inductive biases to derive meaningful attention scores. Yet, current methods often fall short in capturing longer ranges, hierarchical structures, or community structures, which are common …

Graph Classification Graph Regression Node Classification

Orion-Bix: Bi-Axial Attention for Tabular In-Context Learning

2025-11-28 · Mohamed Bouadi, Pratinav Seth, Aditya Tanna, Vinay Kumar Sankarapu arxiv

Tabular data drive most real-world machine learning applications, yet building general-purpose models for them remains difficult. Mixed numeric and categorical fields, weak feature structure, and limited labeled data mak…

Estimating cognitive biases with attention-aware inverse planning

2025-10-29 · Sounak Banerjee, Daphne Cornelisse, Deepak Gopinath, Emily Sumner 외 arxiv

People's goal-directed behaviors are influenced by their cognitive biases, and autonomous systems that interact with people should be aware of this. For example, people's attention to objects in their environment will be…

Reinforcement Learning

Taxonomy-aware Dynamic Motion Generation on Hyperbolic Manifolds

2025-09-25 · Luis Augenstein, Noémie Jaquier, Tamim Asfour, Leonel Rozo arxiv

Human-like motion generation for robots often draws inspiration from biomechanical studies, which often categorize complex human motions into hierarchical taxonomies. While these taxonomies provide rich structural inform…

Interpretable Structure-aware Document Encoders with Hierarchical Attention

2019-02-26 · Khalil Mrini, Claudiu Musat, Michael Baeriswyl, Martin Jaggi

We propose a method to create document representations that reflect their internal structure. We modify Tree-LSTMs to hierarchically merge basic elements such as words and sentences into blocks of increasing complexity. …

Document Classification Sentence Word Embeddings