DeepCOVID-XR: An Artificial Intelligence Algorithm to Detect COVID-19 on Chest Radiographs Trained and Tested on a Large U.S. Clinical Data Set

DeepCOVID-XR: An Artificial Intelligence Algorithm to Detect COVID-19 on Chest Radiographs Trained and Tested on a Large U.S. Clinical Data Set. Wehbe, R. M., Sheng, J., Dutta, S., Chai, S., Dravid, A., Barutcu, S., Wu, Y., Cantrell, D. R., Xiao, N., Allen, B. D., MacNealy, G. A., Savas, H., Agrawal, R., Parekh, N., & Katsaggelos, A. K. Radiology, 299(1):E167–E176, Radiological Society of North America, apr, 2021.

Paper doi abstract bibtex

Background: There are characteristic findings of coronavirus disease 2019 (COVID-19) on chest images. An artificial intelligence (AI) algorithm to detect COVID-19 on chest radiographs might be useful for triage or infection control within a hospital setting, but prior reports have been limited by small data sets, poor data quality, or both. Purpose: To present DeepCOVID-XR, a deep learning AI algorithm to detect COVID-19 on chest radiographs, that was trained and tested on a large clinical data set. Materials and Methods: DeepCOVID-XR is an ensemble of convolutional neural networks developed to detect COVID-19 on frontal chest radiographs, with reverse-transcription polymerase chain reaction test results as the reference standard. The algorithm was trained and validated on 14 788 images (4253 positive for COVID-19) from sites across the Northwestern Memorial Health Care System from February 2020 to April 2020 and was then tested on 2214 images (1192 positive for COVID-19) from a single holdout institution. Performance of the algorithm was compared with interpretations from five experienced thoracic radiologists on 300 random test images using the McNemar test for sensitivity and specificity and the DeLong test for the area under the receiver operating characteristic curve (AUC). Results: A total of 5853 patients (mean age, 58 years 6 19 [standard deviation]; 3101 women) were evaluated across data sets. For the entire test set, accuracy of DeepCOVID-XR was 83%, with an AUC of 0.90. For 300 random test images (134 positive for COVID-19), accuracy of DeepCOVID-XR was 82%, compared with that of individual radiologists (range, 76%–81%) and the consensus of all five radiologists (81%). DeepCOVID-XR had a significantly higher sensitivity (71%) than one radiologist (60%, P , .001) and significantly higher specificity (92%) than two radiologists (75%, P , .001; 84%, P = .009). AUC of DeepCOVID-XR was 0.88 compared with the consensus AUC of 0.85 (P = .13 for comparison). With consensus interpretation as the reference standard, the AUC of DeepCOVID-XR was 0.95 (95% CI: 0.92, 0.98). Conclusion: DeepCOVID-XR, an artificial intelligence algorithm, detected coronavirus disease 2019 on chest radiographs with a performance similar to that of experienced thoracic radiologists in consensus.

@article{Wehbe2021,
abstract = {Background: There are characteristic findings of coronavirus disease 2019 (COVID-19) on chest images. An artificial intelligence (AI) algorithm to detect COVID-19 on chest radiographs might be useful for triage or infection control within a hospital setting, but prior reports have been limited by small data sets, poor data quality, or both. Purpose: To present DeepCOVID-XR, a deep learning AI algorithm to detect COVID-19 on chest radiographs, that was trained and tested on a large clinical data set. Materials and Methods: DeepCOVID-XR is an ensemble of convolutional neural networks developed to detect COVID-19 on frontal chest radiographs, with reverse-transcription polymerase chain reaction test results as the reference standard. The algorithm was trained and validated on 14 788 images (4253 positive for COVID-19) from sites across the Northwestern Memorial Health Care System from February 2020 to April 2020 and was then tested on 2214 images (1192 positive for COVID-19) from a single holdout institution. Performance of the algorithm was compared with interpretations from five experienced thoracic radiologists on 300 random test images using the McNemar test for sensitivity and specificity and the DeLong test for the area under the receiver operating characteristic curve (AUC). Results: A total of 5853 patients (mean age, 58 years 6 19 [standard deviation]; 3101 women) were evaluated across data sets. For the entire test set, accuracy of DeepCOVID-XR was 83%, with an AUC of 0.90. For 300 random test images (134 positive for COVID-19), accuracy of DeepCOVID-XR was 82%, compared with that of individual radiologists (range, 76%–81%) and the consensus of all five radiologists (81%). DeepCOVID-XR had a significantly higher sensitivity (71%) than one radiologist (60%, P , .001) and significantly higher specificity (92%) than two radiologists (75%, P , .001; 84%, P = .009). AUC of DeepCOVID-XR was 0.88 compared with the consensus AUC of 0.85 (P = .13 for comparison). With consensus interpretation as the reference standard, the AUC of DeepCOVID-XR was 0.95 (95% CI: 0.92, 0.98). Conclusion: DeepCOVID-XR, an artificial intelligence algorithm, detected coronavirus disease 2019 on chest radiographs with a performance similar to that of experienced thoracic radiologists in consensus.},
annote = {doi: 10.1148/radiol.2020203511},
author = {Wehbe, Ramsey M. and Sheng, Jiayue and Dutta, Shinjan and Chai, Siyuan and Dravid, Amil and Barutcu, Semih and Wu, Yunan and Cantrell, Donald R. and Xiao, Nicholas and Allen, Bradley D. and MacNealy, Gregory A. and Savas, Hatice and Agrawal, Rishi and Parekh, Nishant and Katsaggelos, Aggelos K.},
doi = {10.1148/radiol.2020203511},
issn = {0033-8419},
journal = {Radiology},
month = {apr},
number = {1},
pages = {E167--E176},
pmid = {33231531},
publisher = {Radiological Society of North America},
title = {{DeepCOVID-XR: An Artificial Intelligence Algorithm to Detect COVID-19 on Chest Radiographs Trained and Tested on a Large U.S. Clinical Data Set}},
url = {https://doi.org/10.1148/radiol.2020203511 http://pubs.rsna.org/doi/10.1148/radiol.2020203511},
volume = {299},
year = {2021}
}

Downloads: 0

{"_id":"bJ2KeASTZqe5ZGAXc","bibbaseid":"wehbe-sheng-dutta-chai-dravid-barutcu-wu-cantrell-etal-deepcovidxranartificialintelligencealgorithmtodetectcovid19onchestradiographstrainedandtestedonalargeusclinicaldataset-2021","author_short":["Wehbe, R. M.","Sheng, J.","Dutta, S.","Chai, S.","Dravid, A.","Barutcu, S.","Wu, Y.","Cantrell, D. R.","Xiao, N.","Allen, B. D.","MacNealy, G. A.","Savas, H.","Agrawal, R.","Parekh, N.","Katsaggelos, A. K."],"bibdata":{"bibtype":"article","type":"article","abstract":"Background: There are characteristic findings of coronavirus disease 2019 (COVID-19) on chest images. An artificial intelligence (AI) algorithm to detect COVID-19 on chest radiographs might be useful for triage or infection control within a hospital setting, but prior reports have been limited by small data sets, poor data quality, or both. Purpose: To present DeepCOVID-XR, a deep learning AI algorithm to detect COVID-19 on chest radiographs, that was trained and tested on a large clinical data set. Materials and Methods: DeepCOVID-XR is an ensemble of convolutional neural networks developed to detect COVID-19 on frontal chest radiographs, with reverse-transcription polymerase chain reaction test results as the reference standard. The algorithm was trained and validated on 14 788 images (4253 positive for COVID-19) from sites across the Northwestern Memorial Health Care System from February 2020 to April 2020 and was then tested on 2214 images (1192 positive for COVID-19) from a single holdout institution. Performance of the algorithm was compared with interpretations from five experienced thoracic radiologists on 300 random test images using the McNemar test for sensitivity and specificity and the DeLong test for the area under the receiver operating characteristic curve (AUC). Results: A total of 5853 patients (mean age, 58 years 6 19 [standard deviation]; 3101 women) were evaluated across data sets. For the entire test set, accuracy of DeepCOVID-XR was 83%, with an AUC of 0.90. For 300 random test images (134 positive for COVID-19), accuracy of DeepCOVID-XR was 82%, compared with that of individual radiologists (range, 76%–81%) and the consensus of all five radiologists (81%). DeepCOVID-XR had a significantly higher sensitivity (71%) than one radiologist (60%, P , .001) and significantly higher specificity (92%) than two radiologists (75%, P , .001; 84%, P = .009). AUC of DeepCOVID-XR was 0.88 compared with the consensus AUC of 0.85 (P = .13 for comparison). With consensus interpretation as the reference standard, the AUC of DeepCOVID-XR was 0.95 (95% CI: 0.92, 0.98). Conclusion: DeepCOVID-XR, an artificial intelligence algorithm, detected coronavirus disease 2019 on chest radiographs with a performance similar to that of experienced thoracic radiologists in consensus.","annote":"doi: 10.1148/radiol.2020203511","author":[{"propositions":[],"lastnames":["Wehbe"],"firstnames":["Ramsey","M."],"suffixes":[]},{"propositions":[],"lastnames":["Sheng"],"firstnames":["Jiayue"],"suffixes":[]},{"propositions":[],"lastnames":["Dutta"],"firstnames":["Shinjan"],"suffixes":[]},{"propositions":[],"lastnames":["Chai"],"firstnames":["Siyuan"],"suffixes":[]},{"propositions":[],"lastnames":["Dravid"],"firstnames":["Amil"],"suffixes":[]},{"propositions":[],"lastnames":["Barutcu"],"firstnames":["Semih"],"suffixes":[]},{"propositions":[],"lastnames":["Wu"],"firstnames":["Yunan"],"suffixes":[]},{"propositions":[],"lastnames":["Cantrell"],"firstnames":["Donald","R."],"suffixes":[]},{"propositions":[],"lastnames":["Xiao"],"firstnames":["Nicholas"],"suffixes":[]},{"propositions":[],"lastnames":["Allen"],"firstnames":["Bradley","D."],"suffixes":[]},{"propositions":[],"lastnames":["MacNealy"],"firstnames":["Gregory","A."],"suffixes":[]},{"propositions":[],"lastnames":["Savas"],"firstnames":["Hatice"],"suffixes":[]},{"propositions":[],"lastnames":["Agrawal"],"firstnames":["Rishi"],"suffixes":[]},{"propositions":[],"lastnames":["Parekh"],"firstnames":["Nishant"],"suffixes":[]},{"propositions":[],"lastnames":["Katsaggelos"],"firstnames":["Aggelos","K."],"suffixes":[]}],"doi":"10.1148/radiol.2020203511","issn":"0033-8419","journal":"Radiology","month":"apr","number":"1","pages":"E167–E176","pmid":"33231531","publisher":"Radiological Society of North America","title":"DeepCOVID-XR: An Artificial Intelligence Algorithm to Detect COVID-19 on Chest Radiographs Trained and Tested on a Large U.S. Clinical Data Set","url":"https://doi.org/10.1148/radiol.2020203511 http://pubs.rsna.org/doi/10.1148/radiol.2020203511","volume":"299","year":"2021","bibtex":"@article{Wehbe2021,\nabstract = {Background: There are characteristic findings of coronavirus disease 2019 (COVID-19) on chest images. An artificial intelligence (AI) algorithm to detect COVID-19 on chest radiographs might be useful for triage or infection control within a hospital setting, but prior reports have been limited by small data sets, poor data quality, or both. Purpose: To present DeepCOVID-XR, a deep learning AI algorithm to detect COVID-19 on chest radiographs, that was trained and tested on a large clinical data set. Materials and Methods: DeepCOVID-XR is an ensemble of convolutional neural networks developed to detect COVID-19 on frontal chest radiographs, with reverse-transcription polymerase chain reaction test results as the reference standard. The algorithm was trained and validated on 14 788 images (4253 positive for COVID-19) from sites across the Northwestern Memorial Health Care System from February 2020 to April 2020 and was then tested on 2214 images (1192 positive for COVID-19) from a single holdout institution. Performance of the algorithm was compared with interpretations from five experienced thoracic radiologists on 300 random test images using the McNemar test for sensitivity and specificity and the DeLong test for the area under the receiver operating characteristic curve (AUC). Results: A total of 5853 patients (mean age, 58 years 6 19 [standard deviation]; 3101 women) were evaluated across data sets. For the entire test set, accuracy of DeepCOVID-XR was 83%, with an AUC of 0.90. For 300 random test images (134 positive for COVID-19), accuracy of DeepCOVID-XR was 82%, compared with that of individual radiologists (range, 76%–81%) and the consensus of all five radiologists (81%). DeepCOVID-XR had a significantly higher sensitivity (71%) than one radiologist (60%, P , .001) and significantly higher specificity (92%) than two radiologists (75%, P , .001; 84%, P = .009). AUC of DeepCOVID-XR was 0.88 compared with the consensus AUC of 0.85 (P = .13 for comparison). With consensus interpretation as the reference standard, the AUC of DeepCOVID-XR was 0.95 (95% CI: 0.92, 0.98). Conclusion: DeepCOVID-XR, an artificial intelligence algorithm, detected coronavirus disease 2019 on chest radiographs with a performance similar to that of experienced thoracic radiologists in consensus.},\nannote = {doi: 10.1148/radiol.2020203511},\nauthor = {Wehbe, Ramsey M. and Sheng, Jiayue and Dutta, Shinjan and Chai, Siyuan and Dravid, Amil and Barutcu, Semih and Wu, Yunan and Cantrell, Donald R. and Xiao, Nicholas and Allen, Bradley D. and MacNealy, Gregory A. and Savas, Hatice and Agrawal, Rishi and Parekh, Nishant and Katsaggelos, Aggelos K.},\ndoi = {10.1148/radiol.2020203511},\nissn = {0033-8419},\njournal = {Radiology},\nmonth = {apr},\nnumber = {1},\npages = {E167--E176},\npmid = {33231531},\npublisher = {Radiological Society of North America},\ntitle = {{DeepCOVID-XR: An Artificial Intelligence Algorithm to Detect COVID-19 on Chest Radiographs Trained and Tested on a Large U.S. Clinical Data Set}},\nurl = {https://doi.org/10.1148/radiol.2020203511 http://pubs.rsna.org/doi/10.1148/radiol.2020203511},\nvolume = {299},\nyear = {2021}\n}\n","author_short":["Wehbe, R. M.","Sheng, J.","Dutta, S.","Chai, S.","Dravid, A.","Barutcu, S.","Wu, Y.","Cantrell, D. R.","Xiao, N.","Allen, B. D.","MacNealy, G. A.","Savas, H.","Agrawal, R.","Parekh, N.","Katsaggelos, A. K."],"key":"Wehbe2021","id":"Wehbe2021","bibbaseid":"wehbe-sheng-dutta-chai-dravid-barutcu-wu-cantrell-etal-deepcovidxranartificialintelligencealgorithmtodetectcovid19onchestradiographstrainedandtestedonalargeusclinicaldataset-2021","role":"author","urls":{"Paper":"https://doi.org/10.1148/radiol.2020203511 http://pubs.rsna.org/doi/10.1148/radiol.2020203511"},"metadata":{"authorlinks":{}}},"bibtype":"article","biburl":"https://sites.northwestern.edu/ivpl/files/2023/06/IVPL_Updated_publications-1.bib","dataSources":["ePKPjG8C6yvpk4mEK","ya2CyA73rpZseyrZ8","37Qfzv6wRptYkoSCL","zFPgsTDAW8aDnb5iN","E6Bth2QB5BYjBMZE7","nbnEjsN7MJhurAK9x","PNQZj6FjzoxxJk4Yi","7FpDWDGJ4KgpDiGfB","bod9ms4MQJHuJgPpp","QR9t5P2cLdJuzhfzK","D8k2SxfC5dKNRFgro","7Dwzbxq93HWrJEhT6","qhF8zxmGcJfvtdeAg","fvDEHD49E2ZRwE3fb","H7crv8NWhZup4d4by","DHqokWsryttGh7pJE","vRJd4wNg9HpoZSMHD","sYxQ6pxFgA59JRhxi","w2WahSbYrbcCKBDsC","XasdXLL99y5rygCmq","3gkSihZQRfAD2KBo3","t5XMbyZbtPBo4wBGS","bEpHM2CtrwW2qE8FP","teJzFLHexaz5AQW5z"],"keywords":[],"search_terms":["deepcovid","artificial","intelligence","algorithm","detect","covid","chest","radiographs","trained","tested","large","clinical","data","set","wehbe","sheng","dutta","chai","dravid","barutcu","wu","cantrell","xiao","allen","macnealy","savas","agrawal","parekh","katsaggelos"],"title":"DeepCOVID-XR: An Artificial Intelligence Algorithm to Detect COVID-19 on Chest Radiographs Trained and Tested on a Large U.S. Clinical Data Set","year":2021}