Reproducibility: A Tragedy of Errors. Allison, D. B., Brown, A. W., George, B. J., & Kaiser, K. A. 530(7588):27–29. ![Reproducibility: A Tragedy of Errors [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper doi abstract bibtex Mistakes in peer-reviewed papers are easy to find but hard to fix, report David B. Allison and colleagues. [Excerpt: Three common errors] As the influential twentieth-century statistician Ronald Fisher (pictured) said: ” To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.” [] [...] Frequent errors, once recognized, can be kept out of the literature with targeted education and policies. Three of the most common are outlined below. These and others are described in depth in an upcoming publication7. [::1. Mistaken design or analysis of cluster-randomized trials] In these studies, all participants in a cluster (for example, a cage, school or hospital) are given the same treatment. The number of clusters (not just the number of individuals) must be incorporated into the analysis. Otherwise, results often seem, falsely, to be statistically significant8, 9. Increasing the number of individuals within clusters can increase power, but the gains are minute compared with increasing clusters. Designs with only one cluster per treatment are not valid as randomized experiments, regardless of how many individuals are included. [::2. Miscalculation in meta-analyses] Effect sizes are often miscalculated when meta-analysts are confronted with incomplete information and do not adapt appropriately. Another problem is confusion about how to calculate the variance of effects. Different study designs and meta-analyses require different approaches. Incorrect or inconsistent choices can change effect sizes, study weighting or the overall conclusions4. [::3. Inappropriate baseline comparisons] In at least six articles, authors tested for changes from the baseline in separate groups; if one was significant and one not, the authors (wrongly) proposed a difference between groups. Rather than comparing 'differences in nominal significance' (the DINS error) differences between groups must be compared directly. For studies comparing two equal-sized groups, the DINS error can inflate the false-positive rate from 5\,% to as much as 50\,% (ref. 10). [] [...]
@article{allisonReproducibilityTragedyErrors2016,
title = {Reproducibility: A Tragedy of Errors},
author = {Allison, David B. and Brown, Andrew W. and George, Brandon J. and Kaiser, Kathryn A.},
date = {2016-02},
journaltitle = {Nature},
volume = {530},
pages = {27--29},
issn = {0028-0836},
doi = {10.1038/530027a},
url = {https://doi.org/10.1038/530027a},
abstract = {Mistakes in peer-reviewed papers are easy to find but hard to fix, report David B. Allison and colleagues.
[Excerpt: Three common errors]
As the influential twentieth-century statistician Ronald Fisher (pictured) said: ” To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.”
[] [...]
Frequent errors, once recognized, can be kept out of the literature with targeted education and policies. Three of the most common are outlined below. These and others are described in depth in an upcoming publication7.
[::1. Mistaken design or analysis of cluster-randomized trials] In these studies, all participants in a cluster (for example, a cage, school or hospital) are given the same treatment. The number of clusters (not just the number of individuals) must be incorporated into the analysis. Otherwise, results often seem, falsely, to be statistically significant8, 9. Increasing the number of individuals within clusters can increase power, but the gains are minute compared with increasing clusters. Designs with only one cluster per treatment are not valid as randomized experiments, regardless of how many individuals are included.
[::2. Miscalculation in meta-analyses] Effect sizes are often miscalculated when meta-analysts are confronted with incomplete information and do not adapt appropriately. Another problem is confusion about how to calculate the variance of effects. Different study designs and meta-analyses require different approaches. Incorrect or inconsistent choices can change effect sizes, study weighting or the overall conclusions4.
[::3. Inappropriate baseline comparisons] In at least six articles, authors tested for changes from the baseline in separate groups; if one was significant and one not, the authors (wrongly) proposed a difference between groups. Rather than comparing 'differences in nominal significance' (the DINS error) differences between groups must be compared directly. For studies comparing two equal-sized groups, the DINS error can inflate the false-positive rate from 5\,\% to as much as 50\,\% (ref. 10).
[] [...]},
keywords = {*imported-from-citeulike-INRMM,~INRMM-MiD:c-13924997,~to-add-doi-URL,bias-correction,cognitive-biases,data-collection-bias,peer-review,post-publication-peer-review,publication-errors,reproducible-research,research-management,science-ethics,statistics,uncertainty,uncertainty-propagation},
number = {7588}
}
Downloads: 0
{"_id":"62sKu4MHhoPC42iBH","bibbaseid":"allison-brown-george-kaiser-reproducibilityatragedyoferrors","authorIDs":[],"author_short":["Allison, D. B.","Brown, A. W.","George, B. J.","Kaiser, K. A."],"bibdata":{"bibtype":"article","type":"article","title":"Reproducibility: A Tragedy of Errors","author":[{"propositions":[],"lastnames":["Allison"],"firstnames":["David","B."],"suffixes":[]},{"propositions":[],"lastnames":["Brown"],"firstnames":["Andrew","W."],"suffixes":[]},{"propositions":[],"lastnames":["George"],"firstnames":["Brandon","J."],"suffixes":[]},{"propositions":[],"lastnames":["Kaiser"],"firstnames":["Kathryn","A."],"suffixes":[]}],"date":"2016-02","journaltitle":"Nature","volume":"530","pages":"27–29","issn":"0028-0836","doi":"10.1038/530027a","url":"https://doi.org/10.1038/530027a","abstract":"Mistakes in peer-reviewed papers are easy to find but hard to fix, report David B. Allison and colleagues. [Excerpt: Three common errors] As the influential twentieth-century statistician Ronald Fisher (pictured) said: ” To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.” [] [...] Frequent errors, once recognized, can be kept out of the literature with targeted education and policies. Three of the most common are outlined below. These and others are described in depth in an upcoming publication7. [::1. Mistaken design or analysis of cluster-randomized trials] In these studies, all participants in a cluster (for example, a cage, school or hospital) are given the same treatment. The number of clusters (not just the number of individuals) must be incorporated into the analysis. Otherwise, results often seem, falsely, to be statistically significant8, 9. Increasing the number of individuals within clusters can increase power, but the gains are minute compared with increasing clusters. Designs with only one cluster per treatment are not valid as randomized experiments, regardless of how many individuals are included. [::2. Miscalculation in meta-analyses] Effect sizes are often miscalculated when meta-analysts are confronted with incomplete information and do not adapt appropriately. Another problem is confusion about how to calculate the variance of effects. Different study designs and meta-analyses require different approaches. Incorrect or inconsistent choices can change effect sizes, study weighting or the overall conclusions4. [::3. Inappropriate baseline comparisons] In at least six articles, authors tested for changes from the baseline in separate groups; if one was significant and one not, the authors (wrongly) proposed a difference between groups. Rather than comparing 'differences in nominal significance' (the DINS error) differences between groups must be compared directly. For studies comparing two equal-sized groups, the DINS error can inflate the false-positive rate from 5\\,% to as much as 50\\,% (ref. 10). [] [...]","keywords":"*imported-from-citeulike-INRMM,~INRMM-MiD:c-13924997,~to-add-doi-URL,bias-correction,cognitive-biases,data-collection-bias,peer-review,post-publication-peer-review,publication-errors,reproducible-research,research-management,science-ethics,statistics,uncertainty,uncertainty-propagation","number":"7588","bibtex":"@article{allisonReproducibilityTragedyErrors2016,\n title = {Reproducibility: A Tragedy of Errors},\n author = {Allison, David B. and Brown, Andrew W. and George, Brandon J. and Kaiser, Kathryn A.},\n date = {2016-02},\n journaltitle = {Nature},\n volume = {530},\n pages = {27--29},\n issn = {0028-0836},\n doi = {10.1038/530027a},\n url = {https://doi.org/10.1038/530027a},\n abstract = {Mistakes in peer-reviewed papers are easy to find but hard to fix, report David B. Allison and colleagues.\n\n[Excerpt: Three common errors]\n\nAs the influential twentieth-century statistician Ronald Fisher (pictured) said: ” To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.”\n\n[] [...]\n\nFrequent errors, once recognized, can be kept out of the literature with targeted education and policies. Three of the most common are outlined below. These and others are described in depth in an upcoming publication7.\n\n[::1. Mistaken design or analysis of cluster-randomized trials] In these studies, all participants in a cluster (for example, a cage, school or hospital) are given the same treatment. The number of clusters (not just the number of individuals) must be incorporated into the analysis. Otherwise, results often seem, falsely, to be statistically significant8, 9. Increasing the number of individuals within clusters can increase power, but the gains are minute compared with increasing clusters. Designs with only one cluster per treatment are not valid as randomized experiments, regardless of how many individuals are included.\n\n[::2. Miscalculation in meta-analyses] Effect sizes are often miscalculated when meta-analysts are confronted with incomplete information and do not adapt appropriately. Another problem is confusion about how to calculate the variance of effects. Different study designs and meta-analyses require different approaches. Incorrect or inconsistent choices can change effect sizes, study weighting or the overall conclusions4.\n\n[::3. Inappropriate baseline comparisons] In at least six articles, authors tested for changes from the baseline in separate groups; if one was significant and one not, the authors (wrongly) proposed a difference between groups. Rather than comparing 'differences in nominal significance' (the DINS error) differences between groups must be compared directly. For studies comparing two equal-sized groups, the DINS error can inflate the false-positive rate from 5\\,\\% to as much as 50\\,\\% (ref. 10).\n\n[] [...]},\n keywords = {*imported-from-citeulike-INRMM,~INRMM-MiD:c-13924997,~to-add-doi-URL,bias-correction,cognitive-biases,data-collection-bias,peer-review,post-publication-peer-review,publication-errors,reproducible-research,research-management,science-ethics,statistics,uncertainty,uncertainty-propagation},\n number = {7588}\n}\n\n","author_short":["Allison, D. B.","Brown, A. W.","George, B. J.","Kaiser, K. A."],"key":"allisonReproducibilityTragedyErrors2016","id":"allisonReproducibilityTragedyErrors2016","bibbaseid":"allison-brown-george-kaiser-reproducibilityatragedyoferrors","role":"author","urls":{"Paper":"https://doi.org/10.1038/530027a"},"keyword":["*imported-from-citeulike-INRMM","~INRMM-MiD:c-13924997","~to-add-doi-URL","bias-correction","cognitive-biases","data-collection-bias","peer-review","post-publication-peer-review","publication-errors","reproducible-research","research-management","science-ethics","statistics","uncertainty","uncertainty-propagation"],"downloads":0},"bibtype":"article","biburl":"https://tmpfiles.org/dl/58794/INRMM.bib","creationDate":"2020-07-02T22:41:00.548Z","downloads":0,"keywords":["*imported-from-citeulike-inrmm","~inrmm-mid:c-13924997","~to-add-doi-url","bias-correction","cognitive-biases","data-collection-bias","peer-review","post-publication-peer-review","publication-errors","reproducible-research","research-management","science-ethics","statistics","uncertainty","uncertainty-propagation"],"search_terms":["reproducibility","tragedy","errors","allison","brown","george","kaiser"],"title":"Reproducibility: A Tragedy of Errors","year":null,"dataSources":["DXuKbcZTirdigFKPF"]}