Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses. Coretta, S., Casillas, J. V., Roessig, S., Franke, M., Ahn, B., Al-Hoorie, A. H., Al-Tamimi, J., Alotaibi, N. E., AlShakhori, M. K., Altmiller, R. M., Arantes, P., Athanasopoulou, A., Baese-Berk, M. M., Bailey, G., Sangma, C. B. A, Beier, E. J., Benavides, G. M., Benker, N., BensonMeyer, E. P., Benway, N. R., Berry, G. M., Bing, L., Bjorndahl, C., Bolyanatz, M., Braver, A., Brown, V. A., Brown, A. M., Brugos, A., Buchanan, E. M., Butlin, T., Buxó-Lugo, A., Caillol, C., Cangemi, F., Carignan, C., Carraturo, S., Caudrelier, T., Chodroff, E., Cohn, M., Cronenberg, J., Crouzet, O., Dagar, E. L., Dawson, C., Diantoro, C. A., Dokovova, M., Drake, S., Du, F., Dubuis, M., Duême, F., Durward, M., Egurtzegi, A., Elsherif, M. M., Esser, J., Ferragne, E., Ferreira, F., Fink, L. K., Finley, S., Foster, K., Foulkes, P., Franzke, R., Frazer-McKee, G., Fromont, R., García, C., Geller, J., Grasso, C. L., Greca, P., Grice, M., Grose-Hodge, M. S., Gully, A. J., Halfacre, C., Hauser, I., Hay, J., Haywood, R., Hellmuth, S., Hilger, A. I., Holliday, N., Hoogland, D., Huang, Y., Hughes, V., Icardo Isasa, A., Ilchovska, Z. G., Jeon, H., Jones, J., Junges, M. N., Kaefer, S., Kaland, C., Kelley, M. C., Kelly, N. E., Kettig, T., Khattab, G., Koolen, R., Krahmer, E., Krajewska, D., Krug, A., Kumar, A. A., Lander, A., Lentz, T. O., Li, W., Li, Y., Lialiou, M., Lima, R. M., Lo, J. J. H., Lopez Otero, J. C., Mackay, B., MacLeod, B., Mallard, M., McConnellogue, C. M., Moroz, G., Murali, M., Nalborczyk, L., Nenadić, F., Nieder, J., Nikolić, D., Nogueira, F. G. S., Offerman, H. M., Passoni, E., Pélissier, M., Perry, S. J., Pfiffner, A. M., Proctor, M., Rhodes, R., Rodríguez, N., Roepke, E., Röer, J. P., Sbacco, L., Scarborough, R., Schaeffler, F., Schleef, E., Schmitz, D., Shiryaev, A., Sóskuthy, M., Spaniol, M., Stanley, J. A., Strickler, A., Tavano, A., Tomaschek, F., Tucker, B. V., Turnbull, R., Ugwuanyi, K. O., Urrestarazu-Porta, I., Van De Vijver, R., Van Engen, K. J., Van Miltenburg, E., Wang, B. X., Warner, N., Wehrle, S., Westerbeek, H., Wiener, S., Winters, S., Wong, S. G., Wood, A., Wottawa, J., Xu, C., Zárate-Sández, G., Zellou, G., Zhang, C., Zhu, J., & Roettger, T. B. Advances in Methods and Practices in Psychological Science, 6(3):1–29, July, 2023.
doi  abstract   bibtex   1 download  
Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis that can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling but also from decisions regarding the quantification of the measured behavior. In this study, we gave the same speech-production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further found little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise, or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system, and calibrate their (un)certainty in their conclusions.
@article{coretta_multidimensional_2023,
	title = {Multidimensional {Signals} and {Analytic} {Flexibility}: {Estimating} {Degrees} of {Freedom} in {Human}-{Speech} {Analyses}},
	volume = {6},
	issn = {2515-2459, 2515-2467},
	doi = {10.1177/25152459231162567},
	abstract = {Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis that can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling but also from decisions regarding the quantification of the measured behavior. In this study, we gave the same speech-production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further found little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise, or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system, and calibrate their (un)certainty in their conclusions.},
	language = {en},
	number = {3},
	journal = {Advances in Methods and Practices in Psychological Science},
	author = {Coretta, Stefano and Casillas, Joseph V. and Roessig, Simon and Franke, Michael and Ahn, Byron and Al-Hoorie, Ali H. and Al-Tamimi, Jalal and Alotaibi, Najd E. and AlShakhori, Mohammed K. and Altmiller, Ruth M. and Arantes, Pablo and Athanasopoulou, Angeliki and Baese-Berk, Melissa M. and Bailey, George and Sangma, Cheman Baira A and Beier, Eleonora J. and Benavides, Gabriela M. and Benker, Nicole and BensonMeyer, Emelia P. and Benway, Nina R. and Berry, Grant M. and Bing, Liwen and Bjorndahl, Christina and Bolyanatz, Mariška and Braver, Aaron and Brown, Violet A. and Brown, Alicia M. and Brugos, Alejna and Buchanan, Erin M. and Butlin, Tanna and Buxó-Lugo, Andrés and Caillol, Coline and Cangemi, Francesco and Carignan, Christopher and Carraturo, Sita and Caudrelier, Tiphaine and Chodroff, Eleanor and Cohn, Michelle and Cronenberg, Johanna and Crouzet, Olivier and Dagar, Erica L. and Dawson, Charlotte and Diantoro, Carissa A. and Dokovova, Marie and Drake, Shiloh and Du, Fengting and Dubuis, Margaux and Duême, Florent and Durward, Matthew and Egurtzegi, Ander and Elsherif, Mahmoud M. and Esser, Janina and Ferragne, Emmanuel and Ferreira, Fernanda and Fink, Lauren K. and Finley, Sara and Foster, Kurtis and Foulkes, Paul and Franzke, Rosa and Frazer-McKee, Gabriel and Fromont, Robert and García, Christina and Geller, Jason and Grasso, Camille L. and Greca, Pia and Grice, Martine and Grose-Hodge, Magdalena S. and Gully, Amelia J. and Halfacre, Caitlin and Hauser, Ivy and Hay, Jen and Haywood, Robert and Hellmuth, Sam and Hilger, Allison I. and Holliday, Nicole and Hoogland, Damar and Huang, Yaqian and Hughes, Vincent and Icardo Isasa, Ane and Ilchovska, Zlatomira G. and Jeon, Hae-Sung and Jones, Jacq and Junges, Mágat N. and Kaefer, Stephanie and Kaland, Constantijn and Kelley, Matthew C. and Kelly, Niamh E. and Kettig, Thomas and Khattab, Ghada and Koolen, Ruud and Krahmer, Emiel and Krajewska, Dorota and Krug, Andreas and Kumar, Abhilasha A. and Lander, Anna and Lentz, Tomas O. and Li, Wanyin and Li, Yanyu and Lialiou, Maria and Lima, Ronaldo M. and Lo, Justin J. H. and Lopez Otero, Julio Cesar and Mackay, Bradley and MacLeod, Bethany and Mallard, Mel and McConnellogue, Carol-Ann Mary and Moroz, George and Murali, Mridhula and Nalborczyk, Ladislas and Nenadić, Filip and Nieder, Jessica and Nikolić, Dušan and Nogueira, Francisco G. S. and Offerman, Heather M. and Passoni, Elisa and Pélissier, Maud and Perry, Scott J. and Pfiffner, Alexandra M. and Proctor, Michael and Rhodes, Ryan and Rodríguez, Nicole and Roepke, Elizabeth and Röer, Jan P. and Sbacco, Lucia and Scarborough, Rebecca and Schaeffler, Felix and Schleef, Erik and Schmitz, Dominic and Shiryaev, Alexander and Sóskuthy, Márton and Spaniol, Malin and Stanley, Joseph A. and Strickler, Alyssa and Tavano, Alessandro and Tomaschek, Fabian and Tucker, Benjamin V. and Turnbull, Rory and Ugwuanyi, Kingsley O. and Urrestarazu-Porta, Iñigo and Van De Vijver, Ruben and Van Engen, Kristin J. and Van Miltenburg, Emiel and Wang, Bruce Xiao and Warner, Natasha and Wehrle, Simon and Westerbeek, Hans and Wiener, Seth and Winters, Stephen and Wong, Sidney G.-J. and Wood, Anna and Wottawa, Jane and Xu, Chenzi and Zárate-Sández, Germán and Zellou, Georgia and Zhang, Cong and Zhu, Jian and Roettger, Timo B.},
	month = jul,
	year = {2023},
	pages = {1--29},
}
Downloads: 1
About
Documentation
Keywords
Search
Users
Terms and Conditions of Use
Privacy Policy
Sitemap
© BibBase.org 2021