Examining Estimates of Intervention Effectiveness Using Sensitivity Analysis. An, C., Braun, H., & Walsh, M. E. Educational Measurement: Issues and Practice, 37(2):45–53, June, 2018. ![Examining Estimates of Intervention Effectiveness Using Sensitivity Analysis [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper doi abstract bibtex Making causal inferences from a quasi-experiment is difficult. Sensitivity analysis approaches to address hidden selection bias thus have gained popularity. This study serves as an introduction to a simple but practical form of sensitivity analysis using Monte Carlo simulation procedures. We examine estimated treatment effects for a school-based support intervention designed to address student strengths and needs in academic and nonacademic areas by leveraging partnerships with community agencies. Middle school (Grades 6–8) statewide standardized test scores in mathematics and English language arts (ELA) were examined for students in a large urban district who participated in City Connects during elementary school. Results showed that the estimated treatment effects in both subjects were reduced slightly with the inclusion of U, a hypothesized unobserved binary variable. However, simulated effects fell within one-sided 90% confidence intervals for original treatment effects, suggesting only a mild sensitivity to hidden bias. Moreover, almost identical estimated treatment effects were observed when the magnitude of the mathematical difference between each pair of the conditional probabilities of U given the treatment indicator Z was the same.
@article{an_examining_2018,
title = {Examining {Estimates} of {Intervention} {Effectiveness} {Using} {Sensitivity} {Analysis}},
volume = {37},
issn = {07311745},
url = {http://doi.wiley.com/10.1111/emip.12176},
doi = {10.1111/emip.12176},
abstract = {Making causal inferences from a quasi-experiment is difficult. Sensitivity analysis approaches to address hidden selection bias thus have gained popularity. This study serves as an introduction to a simple but practical form of sensitivity analysis using Monte Carlo simulation procedures. We examine estimated treatment effects for a school-based support intervention designed to address student strengths and needs in academic and nonacademic areas by leveraging partnerships with community agencies. Middle school (Grades 6–8) statewide standardized test scores in mathematics and English language arts (ELA) were examined for students in a large urban district who participated in City Connects during elementary school. Results showed that the estimated treatment effects in both subjects were reduced slightly with the inclusion of U, a hypothesized unobserved binary variable. However, simulated effects fell within one-sided 90\% confidence intervals for original treatment effects, suggesting only a mild sensitivity to hidden bias. Moreover, almost identical estimated treatment effects were observed when the magnitude of the mathematical difference between each pair of the conditional probabilities of U given the treatment indicator Z was the same.},
language = {en},
number = {2},
urldate = {2019-07-25},
journal = {Educational Measurement: Issues and Practice},
author = {An, Chen and Braun, Henry and Walsh, Mary E.},
month = jun,
year = {2018},
keywords = {academic outcomes, ccnx, elementary aged students, empirical study, longitudinal, middle school aged students, monte carlo simulation, publish, school based intervention, sensitivity analysis, standardized test scores, student support},
pages = {45--53},
}
Downloads: 0
{"_id":"tQjjJ5crxMmHo9tRi","bibbaseid":"an-braun-walsh-examiningestimatesofinterventioneffectivenessusingsensitivityanalysis-2018","authorIDs":[],"author_short":["An, C.","Braun, H.","Walsh, M. E."],"bibdata":{"bibtype":"article","type":"article","title":"Examining Estimates of Intervention Effectiveness Using Sensitivity Analysis","volume":"37","issn":"07311745","url":"http://doi.wiley.com/10.1111/emip.12176","doi":"10.1111/emip.12176","abstract":"Making causal inferences from a quasi-experiment is difficult. Sensitivity analysis approaches to address hidden selection bias thus have gained popularity. This study serves as an introduction to a simple but practical form of sensitivity analysis using Monte Carlo simulation procedures. We examine estimated treatment effects for a school-based support intervention designed to address student strengths and needs in academic and nonacademic areas by leveraging partnerships with community agencies. Middle school (Grades 6–8) statewide standardized test scores in mathematics and English language arts (ELA) were examined for students in a large urban district who participated in City Connects during elementary school. Results showed that the estimated treatment effects in both subjects were reduced slightly with the inclusion of U, a hypothesized unobserved binary variable. However, simulated effects fell within one-sided 90% confidence intervals for original treatment effects, suggesting only a mild sensitivity to hidden bias. Moreover, almost identical estimated treatment effects were observed when the magnitude of the mathematical difference between each pair of the conditional probabilities of U given the treatment indicator Z was the same.","language":"en","number":"2","urldate":"2019-07-25","journal":"Educational Measurement: Issues and Practice","author":[{"propositions":[],"lastnames":["An"],"firstnames":["Chen"],"suffixes":[]},{"propositions":[],"lastnames":["Braun"],"firstnames":["Henry"],"suffixes":[]},{"propositions":[],"lastnames":["Walsh"],"firstnames":["Mary","E."],"suffixes":[]}],"month":"June","year":"2018","keywords":"academic outcomes, ccnx, elementary aged students, empirical study, longitudinal, middle school aged students, monte carlo simulation, publish, school based intervention, sensitivity analysis, standardized test scores, student support","pages":"45–53","bibtex":"@article{an_examining_2018,\n\ttitle = {Examining {Estimates} of {Intervention} {Effectiveness} {Using} {Sensitivity} {Analysis}},\n\tvolume = {37},\n\tissn = {07311745},\n\turl = {http://doi.wiley.com/10.1111/emip.12176},\n\tdoi = {10.1111/emip.12176},\n\tabstract = {Making causal inferences from a quasi-experiment is difficult. Sensitivity analysis approaches to address hidden selection bias thus have gained popularity. This study serves as an introduction to a simple but practical form of sensitivity analysis using Monte Carlo simulation procedures. We examine estimated treatment effects for a school-based support intervention designed to address student strengths and needs in academic and nonacademic areas by leveraging partnerships with community agencies. Middle school (Grades 6–8) statewide standardized test scores in mathematics and English language arts (ELA) were examined for students in a large urban district who participated in City Connects during elementary school. Results showed that the estimated treatment effects in both subjects were reduced slightly with the inclusion of U, a hypothesized unobserved binary variable. However, simulated effects fell within one-sided 90\\% confidence intervals for original treatment effects, suggesting only a mild sensitivity to hidden bias. Moreover, almost identical estimated treatment effects were observed when the magnitude of the mathematical difference between each pair of the conditional probabilities of U given the treatment indicator Z was the same.},\n\tlanguage = {en},\n\tnumber = {2},\n\turldate = {2019-07-25},\n\tjournal = {Educational Measurement: Issues and Practice},\n\tauthor = {An, Chen and Braun, Henry and Walsh, Mary E.},\n\tmonth = jun,\n\tyear = {2018},\n\tkeywords = {academic outcomes, ccnx, elementary aged students, empirical study, longitudinal, middle school aged students, monte carlo simulation, publish, school based intervention, sensitivity analysis, standardized test scores, student support},\n\tpages = {45--53},\n}\n\n","author_short":["An, C.","Braun, H.","Walsh, M. E."],"key":"an_examining_2018","id":"an_examining_2018","bibbaseid":"an-braun-walsh-examiningestimatesofinterventioneffectivenessusingsensitivityanalysis-2018","role":"author","urls":{"Paper":"http://doi.wiley.com/10.1111/emip.12176"},"keyword":["academic outcomes","ccnx","elementary aged students","empirical study","longitudinal","middle school aged students","monte carlo simulation","publish","school based intervention","sensitivity analysis","standardized test scores","student support"],"metadata":{"authorlinks":{}}},"bibtype":"article","biburl":"https://api.zotero.org/groups/2430726/items?key=QlaIDChi2UmTysix4Hi1LsZ5&format=bibtex&limit=100","creationDate":"2020-02-03T13:35:58.708Z","downloads":0,"keywords":["academic outcomes","ccnx","elementary aged students","empirical study","longitudinal","middle school aged students","monte carlo simulation","publish","school based intervention","sensitivity analysis","standardized test scores","student support"],"search_terms":["examining","estimates","intervention","effectiveness","using","sensitivity","analysis","an","braun","walsh"],"title":"Examining Estimates of Intervention Effectiveness Using Sensitivity Analysis","year":2018,"dataSources":["xK4DmLPGcc3E8Tcm7"]}