False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Simmons, J. P., Nelson, L. D., & Simonsohn, U. Psychological Science, 22(11):1359–1366, November, 2011. Publisher: SAGE Publications Inc
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant [link]Paper  doi  abstract   bibtex   
In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
@article{simmons_false-positive_2011,
	title = {False-{Positive} {Psychology}: {Undisclosed} {Flexibility} in {Data} {Collection} and {Analysis} {Allows} {Presenting} {Anything} as {Significant}},
	volume = {22},
	issn = {0956-7976},
	shorttitle = {False-{Positive} {Psychology}},
	url = {https://doi.org/10.1177/0956797611417632},
	doi = {10.1177/0956797611417632},
	abstract = {In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.},
	language = {en},
	number = {11},
	urldate = {2021-11-08},
	journal = {Psychological Science},
	author = {Simmons, Joseph P. and Nelson, Leif D. and Simonsohn, Uri},
	month = nov,
	year = {2011},
	note = {Publisher: SAGE Publications Inc},
	keywords = {disclosure, methodology, motivated reasoning, publication},
	pages = {1359--1366},
}

Downloads: 0