The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “ﬁshing expedition” or “p-hacking” and the research hypothesis was posited ahead of time

The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “ﬁshing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Gelman, A. & Loken, E.
abstract bibtex

Researcher degrees of freedom can lead to a multiple comparisons problem, even in settings where researchers perform only a single analysis on their data. The problem is there can be a large number of potential comparisons when the details of data analysis are highly contingent on data, without the researcher having to perform any conscious procedure of ﬁshing or examining multiple p-values. We discuss in the context of several examples of published papers where data-analysis decisions were theoretically-motivated based on previous literature, but where the details of data selection and analysis were not pre-speciﬁed and, as a result, were contingent on data.

@article{gelman_garden_nodate,
	title = {The garden of forking paths: {Why} multiple comparisons can be a problem, even when there is no “ﬁshing expedition” or “p-hacking” and the research hypothesis was posited ahead of time},
	abstract = {Researcher degrees of freedom can lead to a multiple comparisons problem, even in settings where researchers perform only a single analysis on their data. The problem is there can be a large number of potential comparisons when the details of data analysis are highly contingent on data, without the researcher having to perform any conscious procedure of ﬁshing or examining multiple p-values. We discuss in the context of several examples of published papers where data-analysis decisions were theoretically-motivated based on previous literature, but where the details of data selection and analysis were not pre-speciﬁed and, as a result, were contingent on data.},
	language = {en},
	author = {Gelman, Andrew and Loken, Eric},
	pages = {17},
}

Downloads: 0

{"_id":"kEP4L96Lz3hb6F3EK","bibbaseid":"gelman-loken-thegardenofforkingpathswhymultiplecomparisonscanbeaproblemevenwhenthereisnoshingexpeditionorphackingandtheresearchhypothesiswaspositedaheadoftime","author_short":["Gelman, A.","Loken, E."],"bibdata":{"bibtype":"article","type":"article","title":"The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “ﬁshing expedition” or “p-hacking” and the research hypothesis was posited ahead of time","abstract":"Researcher degrees of freedom can lead to a multiple comparisons problem, even in settings where researchers perform only a single analysis on their data. The problem is there can be a large number of potential comparisons when the details of data analysis are highly contingent on data, without the researcher having to perform any conscious procedure of ﬁshing or examining multiple p-values. We discuss in the context of several examples of published papers where data-analysis decisions were theoretically-motivated based on previous literature, but where the details of data selection and analysis were not pre-speciﬁed and, as a result, were contingent on data.","language":"en","author":[{"propositions":[],"lastnames":["Gelman"],"firstnames":["Andrew"],"suffixes":[]},{"propositions":[],"lastnames":["Loken"],"firstnames":["Eric"],"suffixes":[]}],"pages":"17","bibtex":"@article{gelman_garden_nodate,\n\ttitle = {The garden of forking paths: {Why} multiple comparisons can be a problem, even when there is no “ﬁshing expedition” or “p-hacking” and the research hypothesis was posited ahead of time},\n\tabstract = {Researcher degrees of freedom can lead to a multiple comparisons problem, even in settings where researchers perform only a single analysis on their data. The problem is there can be a large number of potential comparisons when the details of data analysis are highly contingent on data, without the researcher having to perform any conscious procedure of ﬁshing or examining multiple p-values. We discuss in the context of several examples of published papers where data-analysis decisions were theoretically-motivated based on previous literature, but where the details of data selection and analysis were not pre-speciﬁed and, as a result, were contingent on data.},\n\tlanguage = {en},\n\tauthor = {Gelman, Andrew and Loken, Eric},\n\tpages = {17},\n}\n\n","author_short":["Gelman, A.","Loken, E."],"key":"gelman_garden_nodate","id":"gelman_garden_nodate","bibbaseid":"gelman-loken-thegardenofforkingpathswhymultiplecomparisonscanbeaproblemevenwhenthereisnoshingexpeditionorphackingandtheresearchhypothesiswaspositedaheadoftime","role":"author","urls":{},"metadata":{"authorlinks":{}}},"bibtype":"article","biburl":"https://api.zotero.org/groups/4504479/items?key=BfP7bN7FF9dJwtyiLBORewdg&format=bibtex&limit=100","dataSources":["bb9wnMe4gzczbyaCs"],"keywords":[],"search_terms":["garden","forking","paths","multiple","comparisons","problem","even","shing","expedition","hacking","research","hypothesis","posited","ahead","time","gelman","loken"],"title":"The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “ﬁshing expedition” or “p-hacking” and the research hypothesis was posited ahead of time","year":null}