Many Analysts, One Dataset: Making Transparent How Variations in Analytical Choices Affect Results

Many Analysts, One Dataset: Making Transparent How Variations in Analytical Choices Affect Results. Silberzahn, R., Uhlmann, E., Martin, D., Anselmi, P., Aust, F., Awtrey, E., Bahník, ̌St ̌epán, Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M., Dalla Rosa, A., Dam, L., Evans, M., Flores Cervantes, I., Fong, N., Gamez-Djokic, M., Glenz, A., Gordon-McKeon, S., Heaton, T., Eriksson, K., Heene, M., Hofelich Mohr, A., Högden, F., Hui, K., Johannesson, M., Kalodimos, J., Kaszubowski, E., Kennedy, D., Lei, R., Lindsay, T., Liverani, S., Madan, C., Molden, D., Molleman, E., Morey, R., Mulder, L., Nijstad, B., Pope, B., Pope, N., Prenoveau, J., Rink, F., Robusto, E., Roderique, H., Sandberg, A., Schlueter, E., S, F., Sherman, M., Sommer, S. A., Sotak, K., Spain, S., Spörlein, C., Stafford, T., Stefanutti, L., Täuber, S., Ullrich, J., Vianello, M., Wagenmakers, E., Witkowiak, M., Yoon, S., & Nosek, B. 2017.
doi abstract bibtex

Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a non-significant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in analysis of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy by which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective analytic choices influence research results.

@article{silberzahn_many_2017,
  title = {Many Analysts, One Dataset: {{Making}} Transparent How Variations in Analytical Choices Affect Results},
  shorttitle = {Many Analysts, One Dataset},
  abstract = {Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69\%) found a statistically significant positive effect and nine teams (31\%) observed a non-significant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in analysis of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy by which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective analytic choices influence research results.},
  doi = {10.17605/osf.io/qkwst},
  author = {Silberzahn, Raphael and Uhlmann, Eric and Martin, Daniel and Anselmi, Pasquale and Aust, Frederik and Awtrey, Eli and Bahn{\'i}k, {\v S}t{\v e}p{\'a}n and Bai, Feng and Bannard, Colin and Bonnier, Evelina and Carlsson, Rickard and Cheung, Felix and Christensen, Garret and Clay, Russ and Craig, Maureen and Dalla Rosa, Anna and Dam, Lammertjan and Evans, Mathew and Flores Cervantes, Ismael and Fong, Nathan and {Gamez-Djokic}, Monica and Glenz, Andreas and {Gordon-McKeon}, Shauna and Heaton, Tim and Eriksson, Karin and Heene, Moritz and Hofelich Mohr, Alicia and H{\"o}gden, Fabia and Hui, Kent and Johannesson, Magnus and Kalodimos, Jonathan and Kaszubowski, Erikson and Kennedy, Deanna and Lei, Ryan and Lindsay, Thomas and Liverani, Silvia and Madan, Christopher and Molden, Daniel and Molleman, Eric and Morey, Richard and Mulder, Laetitia and Nijstad, Bernard and Pope, Bryson and Pope, Nolan and Prenoveau, Jason and Rink, Floor and Robusto, Egidio and Roderique, Hadiya and Sandberg, Anna and Schlueter, Elmar and S, Felix and Sherman, Martin and Sommer, S. Amy and Sotak, Kristin and Spain, Seth and Sp{\"o}rlein, Christoph and Stafford, Tom and Stefanutti, Luca and T{\"a}uber, Susanne and Ullrich, Johannes and Vianello, Michelangelo and Wagenmakers, Eric-Jan and Witkowiak, Maciej and Yoon, Sangsuk and Nosek, Brian},
  year = {2017}
}

Downloads: 0