False-Positive Psychology. Simmons, J. P., Nelson, L. D., & Simonsohn, U. Psychological Science, 22(11):1359--1366, SAGE Publications, November, 2011.
False-Positive Psychology [link]Paper  doi  abstract   bibtex   
In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process. [Excerpt: Requirements for authors] We propose the following six requirements for authors. [::] Authors must decide the rule for terminating data collection before data collection begins and report this rule in the article [...] [::] Authors must collect at least 20 observations per cell or else provide a compelling cost-of-data-collection justification [...] [::] Authors must list all variables collected in a study. [...] [::] Authors must report all experimental conditions, including failed manipulations. [...] [::] If observations are eliminated, authors must also report what the statistical results are if those observations are included. [...] [::] If an analysis includes a covariate, authors must report the statistical results of the analysis without the covariate. [...] [Guidelines for reviewers] We propose the following four guidelines for reviewers. [::] Reviewers should ensure that authors follow the requirements. [...] [::] Reviewers should be more tolerant of imperfections in results. [...] [::] Reviewers should require authors to demonstrate that their results do not hinge on arbitrary analytic decisions. [...] [::] If justifications of data collection or analysis are not compelling, reviewers should require the authors to conduct an exact replication. [...] [Concluding Remarks] Our goal as scientists is not to publish as many articles as we can, but to discover and disseminate truth. Many of us—and this includes the three authors of this article—often lose sight of this goal, yielding to the pressure to do whatever is justifiable to compile a set of studies that we can publish. This is not driven by a willingness to deceive but by the self-serving interpretation of ambiguity, which enables us to convince ourselves that whichever decisions produced the most publishable outcome must have also been the most appropriate. This article advocates a set of disclosure requirements that imposes minimal costs on authors, readers, and reviewers. These solutions will not rid researchers of publication pressures, but they will limit what authors are able to justify as acceptable to others and to themselves. We should embrace these disclosure requirements as if the credibility of our profession depended on them. Because it does.
@article{citeulike:9928940,
    abstract = {In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.


[Excerpt: Requirements for authors]

We propose the following six requirements for authors.

[::] Authors must decide the rule for terminating data collection before data collection begins and report this rule in the article [...]

[::] Authors must collect at least 20 observations per cell or else provide a compelling cost-of-data-collection justification [...]

[::] Authors must list all variables collected in a study.  [...]

[::] Authors must report all experimental conditions, including failed manipulations.  [...]

[::] If observations are eliminated, authors must also report what the statistical results are if those observations are included.  [...]

[::] If an analysis includes a covariate, authors must report the statistical results of the analysis without the covariate. [...]

[Guidelines for reviewers]

We propose the following four guidelines for reviewers.

[::] Reviewers should ensure that authors follow the requirements.  [...]

[::] Reviewers should be more tolerant of imperfections in results.  [...]

[::] Reviewers should require authors to demonstrate that their results do not hinge on arbitrary analytic decisions.  [...]

[::] If justifications of data collection or analysis are not compelling, reviewers should require the authors to conduct an exact replication.  [...]

[Concluding Remarks]

Our goal as scientists is not to publish as many articles as we can, but to discover and disseminate truth. Many of us—and this includes the three authors of this article—often lose sight of this goal, yielding to the pressure to do whatever is justifiable to compile a set of studies that we can publish. This is not driven by a willingness to deceive but by the self-serving interpretation of ambiguity, which enables us to convince ourselves that whichever decisions produced the most publishable outcome must have also been the most appropriate. This article advocates a set of disclosure requirements that imposes minimal costs on authors, readers, and reviewers. These solutions will not rid researchers of publication pressures, but they will limit what authors are able to justify as acceptable to others and to themselves. We should embrace these disclosure requirements as if the credibility of our profession depended on them. Because it does.},
    author = {Simmons, Joseph P. and Nelson, Leif D. and Simonsohn, Uri},
    citeulike-article-id = {9928940},
    citeulike-linkout-0 = {http://mfkp.org/INRMM/article/9928940},
    citeulike-linkout-1 = {http://dx.doi.org/10.1177/0956797611417632},
    citeulike-linkout-2 = {http://scholar.google.it/scholar?cluster=18405714882620278531},
    citeulike-linkout-3 = {http://dx.doi.org/10.1177/0956797611417632},
    citeulike-linkout-4 = {http://pss.sagepub.com/content/22/11/1359.abstract},
    citeulike-linkout-5 = {http://pss.sagepub.com/content/22/11/1359.full.pdf},
    citeulike-linkout-6 = {http://pss.sagepub.com/cgi/content/abstract/22/11/1359},
    citeulike-linkout-7 = {http://view.ncbi.nlm.nih.gov/pubmed/22006061},
    citeulike-linkout-8 = {http://www.hubmed.org/display.cgi?uids=22006061},
    day = {01},
    doi = {10.1177/0956797611417632},
    issn = {1467-9280},
    journal = {Psychological Science},
    keywords = {check-list, cognitive-biases, communicating-uncertainty, false-positive, p-value, psychology, science-ethics, statistics, uncertainty, validation},
    month = nov,
    number = {11},
    pages = {1359--1366},
    pmid = {22006061},
    posted-at = {2013-01-11 18:44:40},
    priority = {2},
    publisher = {SAGE Publications},
    title = {{False-Positive} Psychology},
    url = {http://mfkp.org/INRMM/article/9928940},
    volume = {22},
    year = {2011}
}
Downloads: 0
About
Documentation
Keywords
Search
Users
Terms and Conditions of Use
Privacy Policy
Sitemap
© BibBase.org 2021