ChatGPT and Software Testing Education: Promises & Perils. Jalil, S., Suzzana Rafi, LaToza, T. D, Moran, K., & Lam, W. arXiv.org, March, 2023. Place: Ithaca Publisher: Cornell University Library, arXiv.org![ChatGPT and Software Testing Education: Promises & Perils [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper abstract bibtex Over the past decade, predictive language modeling for code has proven to be a valuable tool for enabling new forms of automation for developers. More recently, we have seen the advent of general purpose "large language models", based on neural transformer architectures, that have been trained on massive datasets of human written text spanning code and natural language. However, despite the demonstrated representational power of such models, interacting with them has historically been constrained to specific task settings, limiting their general applicability. Many of these limitations were recently overcome with the introduction of ChatGPT, a language model created by OpenAI and trained to operate as a conversational agent, enabling it to answer questions and respond to a wide variety of commands from end users. The introduction of models, such as ChatGPT, has already spurred fervent discussion from educators, ranging from fear that students could use these AI tools to circumvent learning, to excitement about the new types of learning opportunities that they might unlock. However, given the nascent nature of these tools, we currently lack fundamental knowledge related to how well they perform in different educational settings, and the potential promise (or danger) that they might pose to traditional forms of instruction. As such, in this paper, we examine how well ChatGPT performs when tasked with answering common questions in a popular software testing curriculum. Our findings indicate that ChatGPT can provide correct or partially correct answers in 55.6% of cases, provide correct or partially correct explanations of answers in 53.0% of cases, and that prompting the tool in a shared question context leads to a marginally higher rate of correct responses. Based on these findings, we discuss the potential promises and perils related to the use of ChatGPT by students and instructors.
@article{jalil_chatgpt_2023,
title = {{ChatGPT} and {Software} {Testing} {Education}: {Promises} \& {Perils}},
url = {https://www.proquest.com/working-papers/chatgpt-software-testing-education-promises-amp/docview/2774362357/se-2},
abstract = {Over the past decade, predictive language modeling for code has proven to be a valuable tool for enabling new forms of automation for developers. More recently, we have seen the advent of general purpose "large language models", based on neural transformer architectures, that have been trained on massive datasets of human written text spanning code and natural language. However, despite the demonstrated representational power of such models, interacting with them has historically been constrained to specific task settings, limiting their general applicability. Many of these limitations were recently overcome with the introduction of ChatGPT, a language model created by OpenAI and trained to operate as a conversational agent, enabling it to answer questions and respond to a wide variety of commands from end users. The introduction of models, such as ChatGPT, has already spurred fervent discussion from educators, ranging from fear that students could use these AI tools to circumvent learning, to excitement about the new types of learning opportunities that they might unlock. However, given the nascent nature of these tools, we currently lack fundamental knowledge related to how well they perform in different educational settings, and the potential promise (or danger) that they might pose to traditional forms of instruction. As such, in this paper, we examine how well ChatGPT performs when tasked with answering common questions in a popular software testing curriculum. Our findings indicate that ChatGPT can provide correct or partially correct answers in 55.6\% of cases, provide correct or partially correct explanations of answers in 53.0\% of cases, and that prompting the tool in a shared question context leads to a marginally higher rate of correct responses. Based on these findings, we discuss the potential promises and perils related to the use of ChatGPT by students and instructors.},
language = {English},
journal = {arXiv.org},
author = {Jalil, Sajed and {Suzzana Rafi} and LaToza, Thomas D and Moran, Kevin and Lam, Wing},
month = mar,
year = {2023},
note = {Place: Ithaca
Publisher: Cornell University Library, arXiv.org},
keywords = {Chatbots, Learning, Business And Economics--Banking And Finance, Questions, Education, Human-Computer Interaction, Massive data points, Natural language processing, Software Engineering, Software testing, Students},
}
Downloads: 0
{"_id":"afGNKuN5LH7CiBjLN","bibbaseid":"jalil-suzzanarafi-latoza-moran-lam-chatgptandsoftwaretestingeducationpromisesperils-2023","author_short":["Jalil, S.","Suzzana Rafi","LaToza, T. D","Moran, K.","Lam, W."],"bibdata":{"bibtype":"article","type":"article","title":"ChatGPT and Software Testing Education: Promises & Perils","url":"https://www.proquest.com/working-papers/chatgpt-software-testing-education-promises-amp/docview/2774362357/se-2","abstract":"Over the past decade, predictive language modeling for code has proven to be a valuable tool for enabling new forms of automation for developers. More recently, we have seen the advent of general purpose \"large language models\", based on neural transformer architectures, that have been trained on massive datasets of human written text spanning code and natural language. However, despite the demonstrated representational power of such models, interacting with them has historically been constrained to specific task settings, limiting their general applicability. Many of these limitations were recently overcome with the introduction of ChatGPT, a language model created by OpenAI and trained to operate as a conversational agent, enabling it to answer questions and respond to a wide variety of commands from end users. The introduction of models, such as ChatGPT, has already spurred fervent discussion from educators, ranging from fear that students could use these AI tools to circumvent learning, to excitement about the new types of learning opportunities that they might unlock. However, given the nascent nature of these tools, we currently lack fundamental knowledge related to how well they perform in different educational settings, and the potential promise (or danger) that they might pose to traditional forms of instruction. As such, in this paper, we examine how well ChatGPT performs when tasked with answering common questions in a popular software testing curriculum. Our findings indicate that ChatGPT can provide correct or partially correct answers in 55.6% of cases, provide correct or partially correct explanations of answers in 53.0% of cases, and that prompting the tool in a shared question context leads to a marginally higher rate of correct responses. Based on these findings, we discuss the potential promises and perils related to the use of ChatGPT by students and instructors.","language":"English","journal":"arXiv.org","author":[{"propositions":[],"lastnames":["Jalil"],"firstnames":["Sajed"],"suffixes":[]},{"firstnames":[],"propositions":[],"lastnames":["Suzzana Rafi"],"suffixes":[]},{"propositions":[],"lastnames":["LaToza"],"firstnames":["Thomas","D"],"suffixes":[]},{"propositions":[],"lastnames":["Moran"],"firstnames":["Kevin"],"suffixes":[]},{"propositions":[],"lastnames":["Lam"],"firstnames":["Wing"],"suffixes":[]}],"month":"March","year":"2023","note":"Place: Ithaca Publisher: Cornell University Library, arXiv.org","keywords":"Chatbots, Learning, Business And Economics–Banking And Finance, Questions, Education, Human-Computer Interaction, Massive data points, Natural language processing, Software Engineering, Software testing, Students","bibtex":"@article{jalil_chatgpt_2023,\n\ttitle = {{ChatGPT} and {Software} {Testing} {Education}: {Promises} \\& {Perils}},\n\turl = {https://www.proquest.com/working-papers/chatgpt-software-testing-education-promises-amp/docview/2774362357/se-2},\n\tabstract = {Over the past decade, predictive language modeling for code has proven to be a valuable tool for enabling new forms of automation for developers. More recently, we have seen the advent of general purpose \"large language models\", based on neural transformer architectures, that have been trained on massive datasets of human written text spanning code and natural language. However, despite the demonstrated representational power of such models, interacting with them has historically been constrained to specific task settings, limiting their general applicability. Many of these limitations were recently overcome with the introduction of ChatGPT, a language model created by OpenAI and trained to operate as a conversational agent, enabling it to answer questions and respond to a wide variety of commands from end users. The introduction of models, such as ChatGPT, has already spurred fervent discussion from educators, ranging from fear that students could use these AI tools to circumvent learning, to excitement about the new types of learning opportunities that they might unlock. However, given the nascent nature of these tools, we currently lack fundamental knowledge related to how well they perform in different educational settings, and the potential promise (or danger) that they might pose to traditional forms of instruction. As such, in this paper, we examine how well ChatGPT performs when tasked with answering common questions in a popular software testing curriculum. Our findings indicate that ChatGPT can provide correct or partially correct answers in 55.6\\% of cases, provide correct or partially correct explanations of answers in 53.0\\% of cases, and that prompting the tool in a shared question context leads to a marginally higher rate of correct responses. Based on these findings, we discuss the potential promises and perils related to the use of ChatGPT by students and instructors.},\n\tlanguage = {English},\n\tjournal = {arXiv.org},\n\tauthor = {Jalil, Sajed and {Suzzana Rafi} and LaToza, Thomas D and Moran, Kevin and Lam, Wing},\n\tmonth = mar,\n\tyear = {2023},\n\tnote = {Place: Ithaca\nPublisher: Cornell University Library, arXiv.org},\n\tkeywords = {Chatbots, Learning, Business And Economics--Banking And Finance, Questions, Education, Human-Computer Interaction, Massive data points, Natural language processing, Software Engineering, Software testing, Students},\n}\n\n","author_short":["Jalil, S.","Suzzana Rafi","LaToza, T. D","Moran, K.","Lam, W."],"key":"jalil_chatgpt_2023","id":"jalil_chatgpt_2023","bibbaseid":"jalil-suzzanarafi-latoza-moran-lam-chatgptandsoftwaretestingeducationpromisesperils-2023","role":"author","urls":{"Paper":"https://www.proquest.com/working-papers/chatgpt-software-testing-education-promises-amp/docview/2774362357/se-2"},"keyword":["Chatbots","Learning","Business And Economics–Banking And Finance","Questions","Education","Human-Computer Interaction","Massive data points","Natural language processing","Software Engineering","Software testing","Students"],"metadata":{"authorlinks":{}}},"bibtype":"article","biburl":"https://bibbase.org/network/files/srCiZydJWRzXD39Ko","dataSources":["cYu6uhMkeFHgRrEty","hLMh7bwHyFsPNWAEL","LKW3iRvnztCpLNTW7","TLD9JxqHfSQQ4r268","X9BvByJrC3kGJexn8","iovNvcnNYDGJcuMq2","NjZJ5ZmWhTtMZBfje"],"keywords":["chatbots","learning","business and economics–banking and finance","questions","education","human-computer interaction","massive data points","natural language processing","software engineering","software testing","students"],"search_terms":["chatgpt","software","testing","education","promises","perils","jalil","suzzana rafi","latoza","moran","lam"],"title":"ChatGPT and Software Testing Education: Promises & Perils","year":2023}