Can ChatGPT advance software testing intelligence? An experience report on metamorphic testing
While ChatGPT is a well-known artificial intelligence chatbot being used to answer human's questions, one may want to discover its potential in advancing software testing. We examine the capability of ChatGPT in advancing the intelligence of software testing through a case study on metamorphic testing (MT), a state-of-the-art software testing technique. We ask ChatGPT to generate candidates of metamorphic relations (MRs), which are basically necessary properties of the object program and which traditionally require human intelligence to identify. These MR candidates are then evaluated in terms of correctness by domain experts. We show that ChatGPT can be used to generate new correct MRs to test several software systems. Having said that, the majority of MR candidates are either defined vaguely or incorrect, especially for systems that have never been tested with MT. ChatGPT can be used to advance software testing intelligence by proposing MR candidates that can be later adopted for implementing tests; but human intelligence should still inevitably be involved to justify and rectify their correctness.
Code (0)
등록된 구현이 없습니다.
Tasks
Chatbotsoftware testingSimilar Papers 제목 키워드 기반
WIP: Assessing the Effectiveness of ChatGPT in Preparatory Testing Activities
This innovative practice WIP paper describes a research study that explores the integration of ChatGPT into the software testing curriculum and evaluates its effectiveness compared to human-generated testing artifacts. I…
software testingMaximizing Penetration Testing Success with Effective Reconnaissance Techniques using ChatGPT
ChatGPT is a generative pretrained transformer language model created using artificial intelligence implemented as chatbot which can provide very detailed responses to a wide variety of questions. As a very contemporary …
ChatbotLanguage ModellingA Qualitative Study on Using ChatGPT for Software Security: Perception vs. Practicality
Artificial Intelligence (AI) advancements have enabled the development of Large Language Models (LLMs) that can perform a variety of tasks with remarkable semantic understanding and accuracy. ChatGPT is one such LLM that…
Information RetrievalVulnerability DetectionComparing Software Developers with ChatGPT: An Empirical Investigation
The advent of automation in particular Software Engineering (SE) tasks has transitioned from theory to reality. Numerous scholarly articles have documented the successful application of Artificial Intelligence to address…
ArticlesChatbotFairnessManagement+1Testing and Evaluation of Large Language Models: Correctness, Non-Toxicity, and Fairness
Large language models (LLMs), such as ChatGPT, have rapidly penetrated into people's work and daily lives over the past few years, due to their extraordinary conversational skills and intelligence. ChatGPT has become the…
FairnessLanguage ModelingLanguage ModellingLogical Reasoning+2