Assertion-Aware Test Code Summarization with Large Language Models

2025-11-09 · Anamul Haque Mollah, Ahmed Aljohani, Hyunsook Do arxiv

Unit tests often lack concise summaries that convey test intent, especially in auto-generated or poorly documented codebases. Large Language Models (LLMs) offer a promising solution, but their effectiveness depends heavily on how they are prompted. Unlike generic code summarization, test-code summarization poses distinct challenges because test methods validate expected behavior through assertions rather than implementing functionality. This paper presents a new benchmark of 91 real-world Java test cases paired with developer-written summaries and conducts a controlled ablation study to investigate how test code-related components-such as the method under test (MUT), assertion messages, and assertion semantics-affect the performance of LLM-generated test summaries. We evaluate four code LLMs (Codex, Codestral, DeepSeek, and Qwen-Coder) across seven prompt configurations using n-gram metrics (BLEU, ROUGE-L, METEOR), semantic similarity (BERTScore), and LLM-based evaluation. Results show that prompting with assertion semantics improves summary quality by an average of 0.10 points (2.3%) over full MUT context (4.45 vs. 4.35) while requiring fewer input tokens. Codex and Qwen-Coder achieve the highest alignment with human-written summaries, while DeepSeek underperforms despite high lexical overlap. The replication package is publicly available at https://doi.org/10. 5281/zenodo.17067550

📄 PDF Abstract BibTeX arXiv:2511.06227

Code (0)

등록된 구현이 없습니다.

Tasks

Semantic Similarity

ASSERTIFY: Utilizing Large Language Models to Generate Assertions for Production Code

2024-11-25 · Mohammad Jalili Torkamani, Abhinav Sharma, Nikita Mehrotra, Rahul Purandare

Production assertions are statements embedded in the code to help developers validate their assumptions about the code. They assist developers in debugging, provide valuable documentation, and enhance code comprehension.…

Few-Shot Learning Prompt Engineering

Cultural Commonsense Knowledge for Intercultural Dialogues

2024-02-16 · Tuan-Phong Nguyen, Simon Razniewski, Gerhard Weikum

Despite recent progress, large language models (LLMs) still face the challenge of appropriately reacting to the intricacies of social and cultural conventions. This paper presents MANGO, a methodology for distilling high…

Knowledge Distillation Specificity

(Security) Assertions by Large Language Models

2023-06-24 · Rahul Kande, Hammond Pearce, Benjamin Tan, Brendan Dolan-Gavitt 외

The security of computer systems typically relies on a hardware root of trust. As vulnerabilities in hardware can have severe implications on a system, there is a need for techniques to support security verification acti…

Code Generation

Assertion-based QA with Question-Aware Open Information Extraction

2018-01-23 · Zhao Yan, Duyu Tang, Nan Duan, Shujie Liu 외

We present assertion based question answering (ABQA), an open domain question answering task that takes a question and a passage as inputs, and outputs a semi-structured assertion consisting of a subject, a predicate and…

Learning-To-Rank Open-Domain Question Answering Open Information Extraction Question Answering+1

ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining

2021-06-01 · ACL 2021 5 · Alexander R. Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang 외

While online conversations can cover a vast amount of information in many different formats, abstractive text summarization has primarily focused on modeling solely news articles. This research gap is due, in part, to th…

Abstractive Text Summarization Argument Mining Articles Community Question Answering+4