Understanding Code Semantics: An Evaluation of Transformer Models in Summarization
This paper delves into the intricacies of code summarization using advanced transformer-based language models. Through empirical studies, we evaluate the efficacy of code summarization by altering function and variable names to explore whether models truly understand code semantics or merely rely on textual cues. We have also introduced adversaries like dead code and commented code across three programming languages (Python, Javascript, and Java) to further scrutinize the model's understanding. Ultimately, our research aims to offer valuable insights into the inner workings of transformer-based LMs, enhancing their ability to understand code and contributing to more efficient software development practices and maintenance workflows.
Code (1)
Tasks
Code SummarizationSimilar Papers 제목 키워드 기반
Code Summarization with Structure-induced Transformer
Code summarization (CS) is becoming a promising area in recent language understanding, which aims to generate sensible human language automatically for programming language in the format of source code, serving in the mo…
Code SummarizationGraph Neural NetworkNatural Language UnderstandingSource Code SummarizationFriendly Topic Assistant for Transformer Based Abstractive Summarization
Abstractive document summarization is a comprehensive task including document understanding and summary generation, in which area Transformer-based models have achieved the state-of-the-art performance. Compared with Tra…
Abstractive Text SummarizationDocument Summarizationdocument understandingTopic ModelsCP-BCS: Binary Code Summarization Guided by Control Flow Graph and Pseudo Code
Automatically generating function summaries for binaries is an extremely valuable but challenging task, since it involves translating the execution behavior and semantics of the low-level language (assembly code) into hu…
Code SummarizationTopic-Guided Abstractive Text Summarization: a Joint Learning Approach
We introduce a new approach for abstractive text summarization, Topic-Guided Abstractive Summarization, which calibrates long-range dependencies from topic-level features with globally salient content. The idea is to inc…
Abstractive Text SummarizationExtractive SummarizationText SummarizationBinary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models
Binary code summarization, while invaluable for understanding code semantics, is challenging due to its labor-intensive nature. This study delves into the potential of large language models (LLMs) for binary code compreh…
BenchmarkingCode SummarizationGPUSemantic Similarity+1