Testing acoustic scene classifiers using Metamorphic Relations. Moreira, D. D., Furtado, A. P., & Nogueira, S. C. In 2020 IEEE International Conference On Artificial Intelligence Testing (AITest), August, 2020. IEEE. ![Testing acoustic scene classifiers using Metamorphic Relations [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper doi abstract bibtex Context: Artificial Intelligence (AI) applications appear as one of the main demands for the software industry nowadays; within this context, speech recognition and acoustic scene detection and classification achieve near-human performance. However, performing systematic testing on these applications is challenging and very costly if we follow traditional testing methodologies. In this scenario, Metamorphic Testing presents an efficient approach to ensuring the quality of machine learning-based systems. Objective: analyze techniques and applications of metamorphic testing and propose metamorphic relations to perform verification and validation of acoustic scene classifiers. Method: the use of Design Science Research to provide iterative and incremental research development, through which the results achieved in the first cycle of research are presented. Results: in the first design cycle, the use of two metamorphic relations focused on attributes and samples permutation were adopted to verify and validate 6 learning algorithms in an acoustic scene classification system, wherein one of the applied relations, on the random forest algorithm, presented a violation, leading to prediction errors and 2.34% drop in its accuracy in one of the tests performed. In the second design cycle, three new relations based on acoustic variations were proposed to validate the audio attributes, where in all of them the ZCR attribute was more effective to deal with the proposed variations. Conclusion: At the end of the two cycles, our approach has revealed verification flaws and has also proven effective for validation purposes of the systems under test, allowing developers of acoustic scene classification systems to apply them to their learning components, audio extraction processes, and to the test and training dataset.
@inproceedings{MoreiraAPFSCN,
author = {Diogo Dantas Moreira and Ana Paula Furtado and Sidney Carvalho Nogueira},
title = {Testing acoustic scene classifiers using Metamorphic Relations},
doi = {10.1109/aitest49225.2020.00014},
abstract = {Context: Artificial Intelligence (AI) applications appear as one of the main demands for the software industry nowadays; within this context, speech recognition and acoustic scene detection and classification achieve near-human performance. However, performing systematic testing on these applications is challenging and very costly if we follow traditional testing methodologies. In this scenario, Metamorphic Testing presents an efficient approach to ensuring the quality of machine learning-based systems. Objective: analyze techniques and applications of metamorphic testing and propose metamorphic relations to perform verification and validation of acoustic scene classifiers. Method: the use of Design Science Research to provide iterative and incremental research development, through which the results achieved in the first cycle of research are presented. Results: in the first design cycle, the use of two metamorphic relations focused on attributes and samples permutation were adopted to verify and validate 6 learning algorithms in an acoustic scene classification system, wherein one of the applied relations, on the random forest algorithm, presented a violation, leading to prediction errors and 2.34% drop in its accuracy in one of the tests performed. In the second design cycle, three new relations based on acoustic variations were proposed to validate the audio attributes, where in all of them the ZCR attribute was more effective to deal with the proposed variations. Conclusion: At the end of the two cycles, our approach has revealed verification flaws and has also proven effective for validation purposes of the systems under test, allowing developers of acoustic scene classification systems to apply them to their learning components, audio extraction processes, and to the test and training dataset.},
url = {https://doi.org/10.1109/aitest49225.2020.00014},
year = {2020},
month = aug,
publisher = {{IEEE}},
booktitle = {2020 {IEEE} International Conference On Artificial Intelligence Testing ({AITest})}
}
Downloads: 0
{"_id":"mx9pc26xQGd6mKb6j","bibbaseid":"moreira-furtado-nogueira-testingacousticsceneclassifiersusingmetamorphicrelations-2020","authorIDs":["fGjx8FcBLFKAxWGNW"],"author_short":["Moreira, D. D.","Furtado, A. P.","Nogueira, S. C."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","author":[{"firstnames":["Diogo","Dantas"],"propositions":[],"lastnames":["Moreira"],"suffixes":[]},{"firstnames":["Ana","Paula"],"propositions":[],"lastnames":["Furtado"],"suffixes":[]},{"firstnames":["Sidney","Carvalho"],"propositions":[],"lastnames":["Nogueira"],"suffixes":[]}],"title":"Testing acoustic scene classifiers using Metamorphic Relations","doi":"10.1109/aitest49225.2020.00014","abstract":"Context: Artificial Intelligence (AI) applications appear as one of the main demands for the software industry nowadays; within this context, speech recognition and acoustic scene detection and classification achieve near-human performance. However, performing systematic testing on these applications is challenging and very costly if we follow traditional testing methodologies. In this scenario, Metamorphic Testing presents an efficient approach to ensuring the quality of machine learning-based systems. Objective: analyze techniques and applications of metamorphic testing and propose metamorphic relations to perform verification and validation of acoustic scene classifiers. Method: the use of Design Science Research to provide iterative and incremental research development, through which the results achieved in the first cycle of research are presented. Results: in the first design cycle, the use of two metamorphic relations focused on attributes and samples permutation were adopted to verify and validate 6 learning algorithms in an acoustic scene classification system, wherein one of the applied relations, on the random forest algorithm, presented a violation, leading to prediction errors and 2.34% drop in its accuracy in one of the tests performed. In the second design cycle, three new relations based on acoustic variations were proposed to validate the audio attributes, where in all of them the ZCR attribute was more effective to deal with the proposed variations. Conclusion: At the end of the two cycles, our approach has revealed verification flaws and has also proven effective for validation purposes of the systems under test, allowing developers of acoustic scene classification systems to apply them to their learning components, audio extraction processes, and to the test and training dataset.","url":"https://doi.org/10.1109/aitest49225.2020.00014","year":"2020","month":"August","publisher":"IEEE","booktitle":"2020 IEEE International Conference On Artificial Intelligence Testing (AITest)","bibtex":"@inproceedings{MoreiraAPFSCN,\n author = {Diogo Dantas Moreira and Ana Paula Furtado and Sidney Carvalho Nogueira},\n title = {Testing acoustic scene classifiers using Metamorphic Relations},\n doi = {10.1109/aitest49225.2020.00014},\n abstract = {Context: Artificial Intelligence (AI) applications appear as one of the main demands for the software industry nowadays; within this context, speech recognition and acoustic scene detection and classification achieve near-human performance. However, performing systematic testing on these applications is challenging and very costly if we follow traditional testing methodologies. In this scenario, Metamorphic Testing presents an efficient approach to ensuring the quality of machine learning-based systems. Objective: analyze techniques and applications of metamorphic testing and propose metamorphic relations to perform verification and validation of acoustic scene classifiers. Method: the use of Design Science Research to provide iterative and incremental research development, through which the results achieved in the first cycle of research are presented. Results: in the first design cycle, the use of two metamorphic relations focused on attributes and samples permutation were adopted to verify and validate 6 learning algorithms in an acoustic scene classification system, wherein one of the applied relations, on the random forest algorithm, presented a violation, leading to prediction errors and 2.34% drop in its accuracy in one of the tests performed. In the second design cycle, three new relations based on acoustic variations were proposed to validate the audio attributes, where in all of them the ZCR attribute was more effective to deal with the proposed variations. Conclusion: At the end of the two cycles, our approach has revealed verification flaws and has also proven effective for validation purposes of the systems under test, allowing developers of acoustic scene classification systems to apply them to their learning components, audio extraction processes, and to the test and training dataset.},\n url = {https://doi.org/10.1109/aitest49225.2020.00014},\n year = {2020},\n month = aug,\n publisher = {{IEEE}},\n booktitle = {2020 {IEEE} International Conference On Artificial Intelligence Testing ({AITest})}\n}\n\n","author_short":["Moreira, D. D.","Furtado, A. P.","Nogueira, S. C."],"key":"MoreiraAPFSCN","id":"MoreiraAPFSCN","bibbaseid":"moreira-furtado-nogueira-testingacousticsceneclassifiersusingmetamorphicrelations-2020","role":"author","urls":{"Paper":"https://doi.org/10.1109/aitest49225.2020.00014"},"metadata":{"authorlinks":{"moreira, d":"https://www.diogodmoreira.com/"}}},"bibtype":"inproceedings","biburl":"www.diogodmoreira.com/pub_ref.bib","creationDate":"2020-10-06T18:46:37.914Z","downloads":1,"keywords":[],"search_terms":["testing","acoustic","scene","classifiers","using","metamorphic","relations","moreira","furtado","nogueira"],"title":"Testing acoustic scene classifiers using Metamorphic Relations","year":2020,"dataSources":["GR4cYwYoKFQZ98KFM","MeyuQcvWcT7spkgCK"]}