Do Not Trust Build Results at Face Value: An Empirical Study of 30 Million CPAN Builds. Zolfagharinia, M., Adams, B., & Guéhéneuc, Y. In Tan, L. & Hindle, A., editors, Proceedings of the 14<sup>th</sup> International Conference on Mining Software Repositories (MSR), pages 312–322, May, 2017. ACM Press. 10 pages.Paper abstract bibtex Continuous Integration (CI) is a cornerstone of modern quality assurance, providing on-demand builds (compilation and tests) of code changes or software releases. Despite the myriad of CI tools and frameworks, the basic activity of interpreting build results is not straightforward, due to not only the number of builds being performed but also, and especially, due to the phenomenon of build inflation, according to which one code change can be built on dozens of different operating systems, run-time environments and hardware architectures. As existing work mostly ignored this inflation, this paper performs a large-scale empirical study of the impact of OS and run-time environment on build failures on 30 million builds of the CPAN ecosystem's CI environment. We observe the evolution of build failures over time, and investigate the impact of OSes and environments on build failures. We show that distributions may fail differently on different OSes and environments and, thus, that the results of CI require careful filtering and selection to identify reliable failure data.
@INPROCEEDINGS{Zolfagharinia17-MSR-CPANBuilds,
author = {Mahdis Zolfagharinia and Bram Adams and Yann-Ga{\"e}l Gu{\'e}h{\'e}neuc},
title = {Do Not Trust Build Results at Face Value: An Empirical Study of 30 Million {CPAN} Builds},
booktitle = {Proceedings of the 14<sup>{th}</sup> International Conference on Mining Software Repositories ({MSR})},
year = {2017},
month = {May},
editor = {Lin Tan and Abram Hindle},
publisher = {ACM Press},
note = {10 pages.},
abstract = {Continuous Integration (CI) is a cornerstone of modern quality assurance, providing on-demand builds (compilation and tests) of code changes or software releases. Despite the myriad of CI tools and frameworks, the basic activity of interpreting build results is not straightforward, due to not only the number of builds being performed but also, and especially, due to the phenomenon of build inflation, according to which one code change can be built on dozens of different operating systems, run-time environments and hardware architectures. As existing work mostly ignored this inflation, this paper performs a large-scale empirical study of the impact of OS and run-time environment on build failures on 30 million builds of the CPAN ecosystem's CI environment. We observe the evolution of build failures over time, and investigate the impact of OSes and environments on build failures. We show that distributions may fail differently on different OSes and environments and, thus, that the results of CI require careful filtering and selection to identify reliable failure data.},
grant = {NSERC DG and CRC on Multi-language Systems},
keywords = {Understanding program comprehension ; MSR},
kind = {MISA},
language = {english},
url = {http://www.ptidej.net/publications/documents/MSR17b.doc.pdf},
pdf = {http://www.ptidej.net/publications/documents/MSR17b.ppt.pdf},
pages = {312--322}
}
Downloads: 0
{"_id":"8HHnvS56bXRnnZ5HC","bibbaseid":"zolfagharinia-adams-guhneuc-donottrustbuildresultsatfacevalueanempiricalstudyof30millioncpanbuilds-2017","downloads":0,"creationDate":"2018-01-17T20:29:42.172Z","title":"Do Not Trust Build Results at Face Value: An Empirical Study of 30 Million CPAN Builds","author_short":["Zolfagharinia, M.","Adams, B.","Guéhéneuc, Y."],"year":2017,"bibtype":"inproceedings","biburl":"http://www.yann-gael.gueheneuc.net/Work/BibBase/guehene (automatically cleaned).bib","bibdata":{"bibtype":"inproceedings","type":"inproceedings","author":[{"firstnames":["Mahdis"],"propositions":[],"lastnames":["Zolfagharinia"],"suffixes":[]},{"firstnames":["Bram"],"propositions":[],"lastnames":["Adams"],"suffixes":[]},{"firstnames":["Yann-Gaël"],"propositions":[],"lastnames":["Guéhéneuc"],"suffixes":[]}],"title":"Do Not Trust Build Results at Face Value: An Empirical Study of 30 Million CPAN Builds","booktitle":"Proceedings of the 14<sup>th</sup> International Conference on Mining Software Repositories (MSR)","year":"2017","month":"May","editor":[{"firstnames":["Lin"],"propositions":[],"lastnames":["Tan"],"suffixes":[]},{"firstnames":["Abram"],"propositions":[],"lastnames":["Hindle"],"suffixes":[]}],"publisher":"ACM Press","note":"10 pages.","abstract":"Continuous Integration (CI) is a cornerstone of modern quality assurance, providing on-demand builds (compilation and tests) of code changes or software releases. Despite the myriad of CI tools and frameworks, the basic activity of interpreting build results is not straightforward, due to not only the number of builds being performed but also, and especially, due to the phenomenon of build inflation, according to which one code change can be built on dozens of different operating systems, run-time environments and hardware architectures. As existing work mostly ignored this inflation, this paper performs a large-scale empirical study of the impact of OS and run-time environment on build failures on 30 million builds of the CPAN ecosystem's CI environment. We observe the evolution of build failures over time, and investigate the impact of OSes and environments on build failures. We show that distributions may fail differently on different OSes and environments and, thus, that the results of CI require careful filtering and selection to identify reliable failure data.","grant":"NSERC DG and CRC on Multi-language Systems","keywords":"Understanding program comprehension ; MSR","kind":"MISA","language":"english","url":"http://www.ptidej.net/publications/documents/MSR17b.doc.pdf","pdf":"http://www.ptidej.net/publications/documents/MSR17b.ppt.pdf","pages":"312–322","bibtex":"@INPROCEEDINGS{Zolfagharinia17-MSR-CPANBuilds,\n author = {Mahdis Zolfagharinia and Bram Adams and Yann-Ga{\\\"e}l Gu{\\'e}h{\\'e}neuc},\n title = {Do Not Trust Build Results at Face Value: An Empirical Study of 30 Million {CPAN} Builds},\n booktitle = {Proceedings of the 14<sup>{th}</sup> International Conference on Mining Software Repositories ({MSR})},\n year = {2017},\n month = {May},\n editor = {Lin Tan and Abram Hindle},\n publisher = {ACM Press},\n note = {10 pages.},\n abstract = {Continuous Integration (CI) is a cornerstone of modern quality assurance, providing on-demand builds (compilation and tests) of code changes or software releases. Despite the myriad of CI tools and frameworks, the basic activity of interpreting build results is not straightforward, due to not only the number of builds being performed but also, and especially, due to the phenomenon of build inflation, according to which one code change can be built on dozens of different operating systems, run-time environments and hardware architectures. As existing work mostly ignored this inflation, this paper performs a large-scale empirical study of the impact of OS and run-time environment on build failures on 30 million builds of the CPAN ecosystem's CI environment. We observe the evolution of build failures over time, and investigate the impact of OSes and environments on build failures. We show that distributions may fail differently on different OSes and environments and, thus, that the results of CI require careful filtering and selection to identify reliable failure data.},\n grant = {NSERC DG and CRC on Multi-language Systems},\n keywords = {Understanding program comprehension ; MSR},\n kind = {MISA},\n language = {english},\n url = {http://www.ptidej.net/publications/documents/MSR17b.doc.pdf},\n pdf = {http://www.ptidej.net/publications/documents/MSR17b.ppt.pdf},\n pages = {312--322}\n}\n\n","author_short":["Zolfagharinia, M.","Adams, B.","Guéhéneuc, Y."],"editor_short":["Tan, L.","Hindle, A."],"key":"Zolfagharinia17-MSR-CPANBuilds","id":"Zolfagharinia17-MSR-CPANBuilds","bibbaseid":"zolfagharinia-adams-guhneuc-donottrustbuildresultsatfacevalueanempiricalstudyof30millioncpanbuilds-2017","role":"author","urls":{"Paper":"http://www.ptidej.net/publications/documents/MSR17b.doc.pdf"},"keyword":["Understanding program comprehension ; MSR"],"metadata":{"authorlinks":{"gu�h�neuc, y":"https://bibbase.org/show?bib=http%3A%2F%2Fwww.yann-gael.gueheneuc.net%2FWork%2FPublications%2FBiblio%2Fcomplete-bibliography.bib&msg=embed","guéhéneuc, y":"https://bibbase.org/show?bib=http://www.yann-gael.gueheneuc.net/Work/BibBase/guehene%20(automatically%20cleaned).bib"}},"downloads":0,"html":""},"search_terms":["trust","build","results","face","value","empirical","study","million","cpan","builds","zolfagharinia","adams","guéhéneuc"],"keywords":["understanding program comprehension ; msr"],"authorIDs":["AfJhKcg96muyPdu7S","xkviMnkrGBneANvMr"],"dataSources":["Sed98LbBeGaXxenrM","8vn5MSGYWB4fAx9Z4"]}