Wisdom of the expert crowd prediction of response for 3 neurology randomized trials. Atanasov, P., Diamantaras, A., MacPherson, A., Vinarov, E., Benjamin, D. M., Shrier, I., Paul, F., Dirnagl, U., & Kimmelman, J. 95(5):e488–e498, 2020. ![Wisdom of the expert crowd prediction of response for 3 neurology randomized trials [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper doi abstract bibtex Objective To explore the accuracy of combined neurology expert forecasts in predicting primary endpoints for trials. Methods We identi?ed one major randomized trial each in stroke, multiple sclerosis (MS), and amyotrophic lateral sclerosis (ALS) that was closing within 6 months. After recruiting a sample of neurology experts for each disease, we elicited forecasts for the primary endpoint outcomes in the trial placebo and treatment arms. Our main outcome was the accuracy of averaged predictions, measured using ordered Brier scores. Scores were compared against an algorithm that o?ered noncommittal predictions. Results Seventy-one neurology experts participated. Combined forecasts of experts were less accurate than a noncommittal prediction algorithm for the stroke trial (pooled Brier score = 0.340, 95% subjective probability interval [sPI] 0.340 to 0.340 vs 0.185 for the uninformed prediction), and approximately as accurate for the MS study (pooled Brier score = 0.107, 95% con?dence interval [CI] 0.081 to 0.133 vs 0.098 for the noncommittal prediction) and the ALS study (pooled Brier score = 0.090, 95% CI 0.081 to 0.185 vs 0.090). The 95% sPIs of individual predictions contained actual trial outcomes among 44% of experts. Only 18% showed prediction skill exceeding the noncommittal prediction. Independent experts and coinvestigators achieved similar levels of accuracy. Conclusion In this ?rst-of-kind exploratory study, averaged expert judgments rarely outperformed noncommittal forecasts. However, experts at least anticipated the possibility of e?ects observed in trials. Our ?ndings, if replicated in di?erent trial samples, caution against the reliance on simple approaches for combining expert opinion in making research and policy decisions.
@article{atanasov2020neurology,
title = {Wisdom of the expert crowd prediction of response for 3 neurology randomized trials},
volume = {95},
issn = {0028-3878, 1526-632X},
url = {http://www.neurology.org/lookup/doi/10.1212/WNL.0000000000009819},
doi = {10.1212/WNL.0000000000009819},
abstract = {Objective To explore the accuracy of combined neurology expert forecasts in predicting primary endpoints for trials.
Methods We identi?ed one major randomized trial each in stroke, multiple sclerosis ({MS}), and amyotrophic lateral sclerosis ({ALS}) that was closing within 6 months. After recruiting a sample of neurology experts for each disease, we elicited forecasts for the primary endpoint outcomes in the trial placebo and treatment arms. Our main outcome was the accuracy of averaged predictions, measured using ordered Brier scores. Scores were compared against an algorithm that o?ered noncommittal predictions.
Results Seventy-one neurology experts participated. Combined forecasts of experts were less accurate than a noncommittal prediction algorithm for the stroke trial (pooled Brier score = 0.340, 95\% subjective probability interval [{sPI}] 0.340 to 0.340 vs 0.185 for the uninformed prediction), and approximately as accurate for the {MS} study (pooled Brier score = 0.107, 95\% con?dence interval [{CI}] 0.081 to 0.133 vs 0.098 for the noncommittal prediction) and the {ALS} study (pooled Brier score = 0.090, 95\% {CI} 0.081 to 0.185 vs 0.090). The 95\% {sPIs} of individual predictions contained actual trial outcomes among 44\% of experts. Only 18\% showed prediction skill exceeding the noncommittal prediction. Independent experts and coinvestigators achieved similar levels of accuracy.
Conclusion In this ?rst-of-kind exploratory study, averaged expert judgments rarely outperformed noncommittal forecasts. However, experts at least anticipated the possibility of e?ects observed in trials. Our ?ndings, if replicated in di?erent trial samples, caution against the reliance on simple approaches for combining expert opinion in making research and policy decisions.},
pages = {e488--e498},
number = {5},
journaltitle = {Neurology},
shortjournal = {Neurology},
author = {Atanasov, Pavel and Diamantaras, Andreas and {MacPherson}, Amanda and Vinarov, Esther and Benjamin, Daniel M. and Shrier, Ian and Paul, Friedemann and Dirnagl, Ulrich and Kimmelman, Jonathan},
urldate = {2020-08-24},
date = {2020-08-04},
year = {2020},
langid = {english},
file = {Atanasov et al. - 2020 - Wisdom of the expert crowd prediction of response .pdf:C\:\\Users\\benjamin\\Zotero\\storage\\RFM6SQFE\\Atanasov et al. - 2020 - Wisdom of the expert crowd prediction of response .pdf:application/pdf}
}
Downloads: 0
{"_id":"PaNWQFvL2aEXEdpoE","bibbaseid":"atanasov-diamantaras-macpherson-vinarov-benjamin-shrier-paul-dirnagl-etal-wisdomoftheexpertcrowdpredictionofresponsefor3neurologyrandomizedtrials-2020","author_short":["Atanasov, P.","Diamantaras, A.","MacPherson, A.","Vinarov, E.","Benjamin, D. M.","Shrier, I.","Paul, F.","Dirnagl, U.","Kimmelman, J."],"bibdata":{"bibtype":"article","type":"article","title":"Wisdom of the expert crowd prediction of response for 3 neurology randomized trials","volume":"95","issn":"0028-3878, 1526-632X","url":"http://www.neurology.org/lookup/doi/10.1212/WNL.0000000000009819","doi":"10.1212/WNL.0000000000009819","abstract":"Objective To explore the accuracy of combined neurology expert forecasts in predicting primary endpoints for trials. Methods We identi?ed one major randomized trial each in stroke, multiple sclerosis (MS), and amyotrophic lateral sclerosis (ALS) that was closing within 6 months. After recruiting a sample of neurology experts for each disease, we elicited forecasts for the primary endpoint outcomes in the trial placebo and treatment arms. Our main outcome was the accuracy of averaged predictions, measured using ordered Brier scores. Scores were compared against an algorithm that o?ered noncommittal predictions. Results Seventy-one neurology experts participated. Combined forecasts of experts were less accurate than a noncommittal prediction algorithm for the stroke trial (pooled Brier score = 0.340, 95% subjective probability interval [sPI] 0.340 to 0.340 vs 0.185 for the uninformed prediction), and approximately as accurate for the MS study (pooled Brier score = 0.107, 95% con?dence interval [CI] 0.081 to 0.133 vs 0.098 for the noncommittal prediction) and the ALS study (pooled Brier score = 0.090, 95% CI 0.081 to 0.185 vs 0.090). The 95% sPIs of individual predictions contained actual trial outcomes among 44% of experts. Only 18% showed prediction skill exceeding the noncommittal prediction. Independent experts and coinvestigators achieved similar levels of accuracy. Conclusion In this ?rst-of-kind exploratory study, averaged expert judgments rarely outperformed noncommittal forecasts. However, experts at least anticipated the possibility of e?ects observed in trials. Our ?ndings, if replicated in di?erent trial samples, caution against the reliance on simple approaches for combining expert opinion in making research and policy decisions.","pages":"e488–e498","number":"5","journaltitle":"Neurology","shortjournal":"Neurology","author":[{"propositions":[],"lastnames":["Atanasov"],"firstnames":["Pavel"],"suffixes":[]},{"propositions":[],"lastnames":["Diamantaras"],"firstnames":["Andreas"],"suffixes":[]},{"propositions":[],"lastnames":["MacPherson"],"firstnames":["Amanda"],"suffixes":[]},{"propositions":[],"lastnames":["Vinarov"],"firstnames":["Esther"],"suffixes":[]},{"propositions":[],"lastnames":["Benjamin"],"firstnames":["Daniel","M."],"suffixes":[]},{"propositions":[],"lastnames":["Shrier"],"firstnames":["Ian"],"suffixes":[]},{"propositions":[],"lastnames":["Paul"],"firstnames":["Friedemann"],"suffixes":[]},{"propositions":[],"lastnames":["Dirnagl"],"firstnames":["Ulrich"],"suffixes":[]},{"propositions":[],"lastnames":["Kimmelman"],"firstnames":["Jonathan"],"suffixes":[]}],"urldate":"2020-08-24","date":"2020-08-04","year":"2020","langid":"english","file":"Atanasov et al. - 2020 - Wisdom of the expert crowd prediction of response .pdf:C\\:\\\\Users\\\\benjamin\\\\Zotero\\\\storage\\\\RFM6SQFE\\\\Atanasov et al. - 2020 - Wisdom of the expert crowd prediction of response .pdf:application/pdf","bibtex":"@article{atanasov2020neurology,\n\ttitle = {Wisdom of the expert crowd prediction of response for 3 neurology randomized trials},\n\tvolume = {95},\n\tissn = {0028-3878, 1526-632X},\n\turl = {http://www.neurology.org/lookup/doi/10.1212/WNL.0000000000009819},\n\tdoi = {10.1212/WNL.0000000000009819},\n\tabstract = {Objective To explore the accuracy of combined neurology expert forecasts in predicting primary endpoints for trials.\nMethods We identi?ed one major randomized trial each in stroke, multiple sclerosis ({MS}), and amyotrophic lateral sclerosis ({ALS}) that was closing within 6 months. After recruiting a sample of neurology experts for each disease, we elicited forecasts for the primary endpoint outcomes in the trial placebo and treatment arms. Our main outcome was the accuracy of averaged predictions, measured using ordered Brier scores. Scores were compared against an algorithm that o?ered noncommittal predictions.\nResults Seventy-one neurology experts participated. Combined forecasts of experts were less accurate than a noncommittal prediction algorithm for the stroke trial (pooled Brier score = 0.340, 95\\% subjective probability interval [{sPI}] 0.340 to 0.340 vs 0.185 for the uninformed prediction), and approximately as accurate for the {MS} study (pooled Brier score = 0.107, 95\\% con?dence interval [{CI}] 0.081 to 0.133 vs 0.098 for the noncommittal prediction) and the {ALS} study (pooled Brier score = 0.090, 95\\% {CI} 0.081 to 0.185 vs 0.090). The 95\\% {sPIs} of individual predictions contained actual trial outcomes among 44\\% of experts. Only 18\\% showed prediction skill exceeding the noncommittal prediction. Independent experts and coinvestigators achieved similar levels of accuracy.\nConclusion In this ?rst-of-kind exploratory study, averaged expert judgments rarely outperformed noncommittal forecasts. However, experts at least anticipated the possibility of e?ects observed in trials. Our ?ndings, if replicated in di?erent trial samples, caution against the reliance on simple approaches for combining expert opinion in making research and policy decisions.},\n\tpages = {e488--e498},\n\tnumber = {5},\n\tjournaltitle = {Neurology},\n\tshortjournal = {Neurology},\n\tauthor = {Atanasov, Pavel and Diamantaras, Andreas and {MacPherson}, Amanda and Vinarov, Esther and Benjamin, Daniel M. and Shrier, Ian and Paul, Friedemann and Dirnagl, Ulrich and Kimmelman, Jonathan},\n\turldate = {2020-08-24},\n\tdate = {2020-08-04},\n\tyear = {2020},\n\tlangid = {english},\n\tfile = {Atanasov et al. - 2020 - Wisdom of the expert crowd prediction of response .pdf:C\\:\\\\Users\\\\benjamin\\\\Zotero\\\\storage\\\\RFM6SQFE\\\\Atanasov et al. - 2020 - Wisdom of the expert crowd prediction of response .pdf:application/pdf}\n}\n\n","author_short":["Atanasov, P.","Diamantaras, A.","MacPherson, A.","Vinarov, E.","Benjamin, D. M.","Shrier, I.","Paul, F.","Dirnagl, U.","Kimmelman, J."],"bibbaseid":"atanasov-diamantaras-macpherson-vinarov-benjamin-shrier-paul-dirnagl-etal-wisdomoftheexpertcrowdpredictionofresponsefor3neurologyrandomizedtrials-2020","role":"author","urls":{"Paper":"http://www.neurology.org/lookup/doi/10.1212/WNL.0000000000009819"},"metadata":{"authorlinks":{}}},"bibtype":"article","biburl":"https://bibbase.org/f/YFikaQA5BbiqYRTEc/galstyan-isi-edu.bib","dataSources":["FCj56B2ST4tyFypkR","X69gAvDa3qySitobN"],"keywords":[],"search_terms":["wisdom","expert","crowd","prediction","response","neurology","randomized","trials","atanasov","diamantaras","macpherson","vinarov","benjamin","shrier","paul","dirnagl","kimmelman"],"title":"Wisdom of the expert crowd prediction of response for 3 neurology randomized trials","year":2020}