Neural Keyphrase Generation: Analysis and Evaluation
Keyphrase generation aims at generating topical phrases from a given text either by copying from the original text (present keyphrases) or by producing new keyphrases (absent keyphrases) that capture the semantic meaning of the text. Encoder-decoder models are most widely used for this task because of their capabilities for absent keyphrase generation. However, there has been little to no analysis on the performance and behavior of such models for keyphrase generation. In this paper, we study various tendencies exhibited by three strong models: T5 (based on a pre-trained transformer), CatSeq-Transformer (a non-pretrained Transformer), and ExHiRD (based on a recurrent neural network). We analyze prediction confidence scores, model calibration, and the effect of token position on keyphrases generation. Moreover, we motivate and propose a novel metric framework, SoftKeyScore, to evaluate the similarity between two sets of keyphrases by using softscores to account for partial matching and semantic similarity. We find that SoftKeyScore is more suitable than the standard F1 metric for evaluating two sets of given keyphrases.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderKeyphrase GenerationSemantic SimilaritySemantic Textual SimilarityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Neural Keyphrase Generation: Analysis and Evaluation
Keyphrase generation aims at generating topical phrases from a given text either by copying from the original text (present keyphrases) or by producing new keyphrases (absent keyphrases) that capture the semantic meaning…
DecoderKeyphrase GenerationSemantic SimilaritySemantic Textual SimilarityAn Analysis of Datasets, Metrics and Models in Keyphrase Generation
Keyphrase generation refers to the task of producing a set of words or phrases that summarises the content of a document. Continuous efforts have been dedicated to this task over the past few years, spreading across mult…
Keyphrase GenerationOne Size Does Not Fit All: Generating and Evaluating Variable Number of Keyphrases
Different texts shall by nature correspond to different number of keyphrases. This desideratum is largely missing from existing neural keyphrase generation models. In this study, we address this problem from both modelin…
AllDecoderDiversityKeyphrase GenerationIs ChatGPT A Good Keyphrase Generator? A Preliminary Study
The emergence of ChatGPT has recently garnered significant attention from the computational linguistics community. To demonstrate its capabilities as a keyphrase generator, we conduct a preliminary evaluation of ChatGPT …
Diversitydocument understandingKeyphrase GenerationNeural Keyphrase Generation via Reinforcement Learning with Adaptive Rewards
Generating keyphrases that summarize the main points of a document is a fundamental task in natural language processing. Although existing generative models are capable of predicting multiple keyphrases for an input docu…
Keyphrase Generationreinforcement-learningReinforcement LearningReinforcement Learning (RL)