Issues and Advances in Anomaly Detection Evaluation for Joint Human-Automated Systems. Rieth, C. A., Amsel, B. D., Tran, R., & Cook, M. B. In Advances in Human Factors in Robots and Unmanned Systems, of Advances in Intelligent Systems and Computing, pages 52–63, Cham, 2018. Springer International Publishing. doi abstract bibtex As human-managed systems become more complex, automated anomaly detection can provide assistance—but only if it is effective. Rigorous evaluation of automated detection is vital for determining its effectiveness before implementation into systems. We identified recurring issues in evaluation practices limiting the conclusions that can be applied from published studies to broader application. In this paper, we demonstrate the implications of these issues and illustrate solutions. We show how receiver operating characteristic curves can reveal performance tradeoffs masked by reporting of single metric results and how using multiple simulation data examples can prevent biases that result from evaluation using single training and testing examples. We also provide methods for incorporating detection latency into tradeoff analyses. Application of these methods will help to provide researchers, engineers, and decision makers with a more objective basis for anomaly detection performance evaluation, resulting in greater utility, better performance, and cost savings in systems engineering.
@inproceedings{rieth_issues_2018,
address = {Cham},
series = {Advances in {Intelligent} {Systems} and {Computing}},
title = {Issues and {Advances} in {Anomaly} {Detection} {Evaluation} for {Joint} {Human}-{Automated} {Systems}},
isbn = {978-3-319-60384-1},
doi = {10.1007/978-3-319-60384-1_6},
abstract = {As human-managed systems become more complex, automated anomaly detection can provide assistance—but only if it is effective. Rigorous evaluation of automated detection is vital for determining its effectiveness before implementation into systems. We identified recurring issues in evaluation practices limiting the conclusions that can be applied from published studies to broader application. In this paper, we demonstrate the implications of these issues and illustrate solutions. We show how receiver operating characteristic curves can reveal performance tradeoffs masked by reporting of single metric results and how using multiple simulation data examples can prevent biases that result from evaluation using single training and testing examples. We also provide methods for incorporating detection latency into tradeoff analyses. Application of these methods will help to provide researchers, engineers, and decision makers with a more objective basis for anomaly detection performance evaluation, resulting in greater utility, better performance, and cost savings in systems engineering.},
language = {en},
booktitle = {Advances in {Human} {Factors} in {Robots} and {Unmanned} {Systems}},
publisher = {Springer International Publishing},
author = {Rieth, Cory A. and Amsel, Ben D. and Tran, Randy and Cook, Maia B.},
editor = {Chen, Jessie},
year = {2018},
keywords = {Anomaly detection, Automation evaluation, Receiver operating characteristic, Tennessee Eastman process simulation},
pages = {52--63},
}
Downloads: 0
{"_id":"kQjDe9ozpSD89EPJX","bibbaseid":"rieth-amsel-tran-cook-issuesandadvancesinanomalydetectionevaluationforjointhumanautomatedsystems-2018","author_short":["Rieth, C. A.","Amsel, B. D.","Tran, R.","Cook, M. B."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","address":"Cham","series":"Advances in Intelligent Systems and Computing","title":"Issues and Advances in Anomaly Detection Evaluation for Joint Human-Automated Systems","isbn":"978-3-319-60384-1","doi":"10.1007/978-3-319-60384-1_6","abstract":"As human-managed systems become more complex, automated anomaly detection can provide assistance—but only if it is effective. Rigorous evaluation of automated detection is vital for determining its effectiveness before implementation into systems. We identified recurring issues in evaluation practices limiting the conclusions that can be applied from published studies to broader application. In this paper, we demonstrate the implications of these issues and illustrate solutions. We show how receiver operating characteristic curves can reveal performance tradeoffs masked by reporting of single metric results and how using multiple simulation data examples can prevent biases that result from evaluation using single training and testing examples. We also provide methods for incorporating detection latency into tradeoff analyses. Application of these methods will help to provide researchers, engineers, and decision makers with a more objective basis for anomaly detection performance evaluation, resulting in greater utility, better performance, and cost savings in systems engineering.","language":"en","booktitle":"Advances in Human Factors in Robots and Unmanned Systems","publisher":"Springer International Publishing","author":[{"propositions":[],"lastnames":["Rieth"],"firstnames":["Cory","A."],"suffixes":[]},{"propositions":[],"lastnames":["Amsel"],"firstnames":["Ben","D."],"suffixes":[]},{"propositions":[],"lastnames":["Tran"],"firstnames":["Randy"],"suffixes":[]},{"propositions":[],"lastnames":["Cook"],"firstnames":["Maia","B."],"suffixes":[]}],"editor":[{"propositions":[],"lastnames":["Chen"],"firstnames":["Jessie"],"suffixes":[]}],"year":"2018","keywords":"Anomaly detection, Automation evaluation, Receiver operating characteristic, Tennessee Eastman process simulation","pages":"52–63","bibtex":"@inproceedings{rieth_issues_2018,\n\taddress = {Cham},\n\tseries = {Advances in {Intelligent} {Systems} and {Computing}},\n\ttitle = {Issues and {Advances} in {Anomaly} {Detection} {Evaluation} for {Joint} {Human}-{Automated} {Systems}},\n\tisbn = {978-3-319-60384-1},\n\tdoi = {10.1007/978-3-319-60384-1_6},\n\tabstract = {As human-managed systems become more complex, automated anomaly detection can provide assistance—but only if it is effective. Rigorous evaluation of automated detection is vital for determining its effectiveness before implementation into systems. We identified recurring issues in evaluation practices limiting the conclusions that can be applied from published studies to broader application. In this paper, we demonstrate the implications of these issues and illustrate solutions. We show how receiver operating characteristic curves can reveal performance tradeoffs masked by reporting of single metric results and how using multiple simulation data examples can prevent biases that result from evaluation using single training and testing examples. We also provide methods for incorporating detection latency into tradeoff analyses. Application of these methods will help to provide researchers, engineers, and decision makers with a more objective basis for anomaly detection performance evaluation, resulting in greater utility, better performance, and cost savings in systems engineering.},\n\tlanguage = {en},\n\tbooktitle = {Advances in {Human} {Factors} in {Robots} and {Unmanned} {Systems}},\n\tpublisher = {Springer International Publishing},\n\tauthor = {Rieth, Cory A. and Amsel, Ben D. and Tran, Randy and Cook, Maia B.},\n\teditor = {Chen, Jessie},\n\tyear = {2018},\n\tkeywords = {Anomaly detection, Automation evaluation, Receiver operating characteristic, Tennessee Eastman process simulation},\n\tpages = {52--63},\n}\n\n","author_short":["Rieth, C. A.","Amsel, B. D.","Tran, R.","Cook, M. B."],"editor_short":["Chen, J."],"key":"rieth_issues_2018","id":"rieth_issues_2018","bibbaseid":"rieth-amsel-tran-cook-issuesandadvancesinanomalydetectionevaluationforjointhumanautomatedsystems-2018","role":"author","urls":{},"keyword":["Anomaly detection","Automation evaluation","Receiver operating characteristic","Tennessee Eastman process simulation"],"metadata":{"authorlinks":{}}},"bibtype":"inproceedings","biburl":"https://bibbase.org/zotero/mh_lenguyen","dataSources":["iwKepCrWBps7ojhDx"],"keywords":["anomaly detection","automation evaluation","receiver operating characteristic","tennessee eastman process simulation"],"search_terms":["issues","advances","anomaly","detection","evaluation","joint","human","automated","systems","rieth","amsel","tran","cook"],"title":"Issues and Advances in Anomaly Detection Evaluation for Joint Human-Automated Systems","year":2018}