TopK Copy

2000년 도입 · 논문 1편에서 사용

TopK Copy is a cross-attention guided copy mechanism for entity extraction where only the Top-$k$ important attention heads are used for computing copy distributions. The motivation is that that attention heads may not equally important, and that some heads can be pruned out with a marginal decrease in overall performance. Attention probabilities produced by insignificant attention heads may be noisy. Thus, computing copy distributions without these heads could improve the model’s ability to infer the importance of each token in the input document.

출처: Document-level Entity-based Extraction as Template Generation

소개 논문: Document-level Entity-based Extraction as Template Generation