Filling a gap in safety metrics: development of a patient-centred framework to identify and categorise patient-reported breakdowns related to the diagnostic process in ambulatory care. Bell, S. K, Bourgeois, F., DesRoches, C. M, Dong, J., Harcourt, K., Liu, S. K, Lowe, E., McGaffigan, P., Ngo, L. H, Novack, S. A, Ralston, J. D, Salmi, L., Schrandt, S., Sheridan, S., Sokol-Hessner, L., Thomas, G., & Thomas, E. J BMJ Quality & Safety, 31(7):526–540, July, 2022.
Paper doi abstract bibtex Background Patients and families are important contributors to the diagnostic team, but their perspectives are not reflected in current diagnostic measures. Patients/families can identify some breakdowns in the diagnostic process beyond the clinician’s view. We aimed to develop a framework with patients/families to help organisations identify and categorise patient-reported diagnostic process-related breakdowns (PRDBs) to inform organisational learning. Method A multi-stakeholder advisory group including patients, families, clinicians, and experts in diagnostic error, patient engagement and safety, and user-centred design, co-developed a framework for PRDBs in ambulatory care. We tested the framework using standard qualitative analysis methods with two physicians and one patient coder, analysing 2165 patient-reported ambulatory errors in two large surveys representing 25 425 US respondents. We tested intercoder reliability of breakdown categorisation using the Gwet’s AC1 and Cohen’s kappa statistic. We considered agreement coefficients 0.61–0.8=good agreement and 0.81–1.00=excellent agreement. Results The framework describes 7 patient-reported breakdown categories (with 40 subcategories), 19 patient-identified contributing factors and 11 potential patient-reported impacts. Patients identified breakdowns in each step of the diagnostic process, including missing or inaccurate main concerns and symptoms; missing/outdated test results; and communication breakdowns such as not feeling heard or misalignment between patient and provider about symptoms, events, or their significance. The frequency of PRDBs was 6.4% in one dataset and 6.9% in the other. Intercoder reliability showed good-to-excellent reliability in each dataset: AC1 0.89 (95% CI 0.89 to 0.90) to 0.96 (95% CI 0.95 to 0.97); kappa 0.64 (95% CI 0.62, to 0.66) to 0.85 (95% CI 0.83 to 0.88). Conclusions The PRDB framework, developed in partnership with patients/families, can help organisations identify and reliably categorise PRDBs, including some that are invisible to clinicians; guide interventions to engage patients and families as diagnostic partners; and inform whole organisational learning.
@article{bell_filling_2022,
title = {Filling a gap in safety metrics: development of a patient-centred framework to identify and categorise patient-reported breakdowns related to the diagnostic process in ambulatory care},
volume = {31},
issn = {2044-5415, 2044-5423},
shorttitle = {Filling a gap in safety metrics},
url = {https://qualitysafety.bmj.com/lookup/doi/10.1136/bmjqs-2021-013672},
doi = {10.1136/bmjqs-2021-013672},
abstract = {Background
Patients and families are important contributors to the diagnostic team, but their perspectives are not reflected in current diagnostic measures. Patients/families can identify some breakdowns in the diagnostic process beyond the clinician’s view. We aimed to develop a framework with patients/families to help organisations identify and categorise patient-reported diagnostic process-related breakdowns (PRDBs) to inform organisational learning.
Method
A multi-stakeholder advisory group including patients, families, clinicians, and experts in diagnostic error, patient engagement and safety, and user-centred design, co-developed a framework for PRDBs in ambulatory care. We tested the framework using standard qualitative analysis methods with two physicians and one patient coder, analysing 2165 patient-reported ambulatory errors in two large surveys representing 25 425 US respondents. We tested intercoder reliability of breakdown categorisation using the Gwet’s AC1 and Cohen’s kappa statistic. We considered agreement coefficients 0.61–0.8=good agreement and 0.81–1.00=excellent agreement.
Results
The framework describes 7 patient-reported breakdown categories (with 40 subcategories), 19 patient-identified contributing factors and 11 potential patient-reported impacts. Patients identified breakdowns in each step of the diagnostic process, including missing or inaccurate main concerns and symptoms; missing/outdated test results; and communication breakdowns such as not feeling heard or misalignment between patient and provider about symptoms, events, or their significance. The frequency of PRDBs was 6.4\% in one dataset and 6.9\% in the other. Intercoder reliability showed good-to-excellent reliability in each dataset: AC1 0.89 (95\% CI 0.89 to 0.90) to 0.96 (95\% CI 0.95 to 0.97); kappa 0.64 (95\% CI 0.62, to 0.66) to 0.85 (95\% CI 0.83 to 0.88).
Conclusions
The PRDB framework, developed in partnership with patients/families, can help organisations identify and reliably categorise PRDBs, including some that are invisible to clinicians; guide interventions to engage patients and families as diagnostic partners; and inform whole organisational learning.},
language = {en},
number = {7},
urldate = {2026-08-04},
journal = {BMJ Quality \& Safety},
author = {Bell, Sigall K and Bourgeois, Fabienne and DesRoches, Catherine M and Dong, Joe and Harcourt, Kendall and Liu, Stephen K and Lowe, Elizabeth and McGaffigan, Patricia and Ngo, Long H and Novack, Sandy A and Ralston, James D and Salmi, Liz and Schrandt, Suz and Sheridan, Sue and Sokol-Hessner, Lauge and Thomas, Glenda and Thomas, Eric J},
month = jul,
year = {2022},
pages = {526--540},
}
Downloads: 0
{"_id":"mjvpr9kyW7YsL3Sxk","bibbaseid":"bell-bourgeois-desroches-dong-harcourt-liu-lowe-mcgaffigan-etal-fillingagapinsafetymetricsdevelopmentofapatientcentredframeworktoidentifyandcategorisepatientreportedbreakdownsrelatedtothediagnosticprocessinambulatorycare-2022","author_short":["Bell, S. K","Bourgeois, F.","DesRoches, C. M","Dong, J.","Harcourt, K.","Liu, S. K","Lowe, E.","McGaffigan, P.","Ngo, L. H","Novack, S. A","Ralston, J. D","Salmi, L.","Schrandt, S.","Sheridan, S.","Sokol-Hessner, L.","Thomas, G.","Thomas, E. J"],"bibdata":{"bibtype":"article","type":"article","title":"Filling a gap in safety metrics: development of a patient-centred framework to identify and categorise patient-reported breakdowns related to the diagnostic process in ambulatory care","volume":"31","issn":"2044-5415, 2044-5423","shorttitle":"Filling a gap in safety metrics","url":"https://qualitysafety.bmj.com/lookup/doi/10.1136/bmjqs-2021-013672","doi":"10.1136/bmjqs-2021-013672","abstract":"Background Patients and families are important contributors to the diagnostic team, but their perspectives are not reflected in current diagnostic measures. Patients/families can identify some breakdowns in the diagnostic process beyond the clinician’s view. We aimed to develop a framework with patients/families to help organisations identify and categorise patient-reported diagnostic process-related breakdowns (PRDBs) to inform organisational learning. Method A multi-stakeholder advisory group including patients, families, clinicians, and experts in diagnostic error, patient engagement and safety, and user-centred design, co-developed a framework for PRDBs in ambulatory care. We tested the framework using standard qualitative analysis methods with two physicians and one patient coder, analysing 2165 patient-reported ambulatory errors in two large surveys representing 25 425 US respondents. We tested intercoder reliability of breakdown categorisation using the Gwet’s AC1 and Cohen’s kappa statistic. We considered agreement coefficients 0.61–0.8=good agreement and 0.81–1.00=excellent agreement. Results The framework describes 7 patient-reported breakdown categories (with 40 subcategories), 19 patient-identified contributing factors and 11 potential patient-reported impacts. Patients identified breakdowns in each step of the diagnostic process, including missing or inaccurate main concerns and symptoms; missing/outdated test results; and communication breakdowns such as not feeling heard or misalignment between patient and provider about symptoms, events, or their significance. The frequency of PRDBs was 6.4% in one dataset and 6.9% in the other. Intercoder reliability showed good-to-excellent reliability in each dataset: AC1 0.89 (95% CI 0.89 to 0.90) to 0.96 (95% CI 0.95 to 0.97); kappa 0.64 (95% CI 0.62, to 0.66) to 0.85 (95% CI 0.83 to 0.88). Conclusions The PRDB framework, developed in partnership with patients/families, can help organisations identify and reliably categorise PRDBs, including some that are invisible to clinicians; guide interventions to engage patients and families as diagnostic partners; and inform whole organisational learning.","language":"en","number":"7","urldate":"2026-08-04","journal":"BMJ Quality & Safety","author":[{"propositions":[],"lastnames":["Bell"],"firstnames":["Sigall","K"],"suffixes":[]},{"propositions":[],"lastnames":["Bourgeois"],"firstnames":["Fabienne"],"suffixes":[]},{"propositions":[],"lastnames":["DesRoches"],"firstnames":["Catherine","M"],"suffixes":[]},{"propositions":[],"lastnames":["Dong"],"firstnames":["Joe"],"suffixes":[]},{"propositions":[],"lastnames":["Harcourt"],"firstnames":["Kendall"],"suffixes":[]},{"propositions":[],"lastnames":["Liu"],"firstnames":["Stephen","K"],"suffixes":[]},{"propositions":[],"lastnames":["Lowe"],"firstnames":["Elizabeth"],"suffixes":[]},{"propositions":[],"lastnames":["McGaffigan"],"firstnames":["Patricia"],"suffixes":[]},{"propositions":[],"lastnames":["Ngo"],"firstnames":["Long","H"],"suffixes":[]},{"propositions":[],"lastnames":["Novack"],"firstnames":["Sandy","A"],"suffixes":[]},{"propositions":[],"lastnames":["Ralston"],"firstnames":["James","D"],"suffixes":[]},{"propositions":[],"lastnames":["Salmi"],"firstnames":["Liz"],"suffixes":[]},{"propositions":[],"lastnames":["Schrandt"],"firstnames":["Suz"],"suffixes":[]},{"propositions":[],"lastnames":["Sheridan"],"firstnames":["Sue"],"suffixes":[]},{"propositions":[],"lastnames":["Sokol-Hessner"],"firstnames":["Lauge"],"suffixes":[]},{"propositions":[],"lastnames":["Thomas"],"firstnames":["Glenda"],"suffixes":[]},{"propositions":[],"lastnames":["Thomas"],"firstnames":["Eric","J"],"suffixes":[]}],"month":"July","year":"2022","pages":"526–540","bibtex":"@article{bell_filling_2022,\n\ttitle = {Filling a gap in safety metrics: development of a patient-centred framework to identify and categorise patient-reported breakdowns related to the diagnostic process in ambulatory care},\n\tvolume = {31},\n\tissn = {2044-5415, 2044-5423},\n\tshorttitle = {Filling a gap in safety metrics},\n\turl = {https://qualitysafety.bmj.com/lookup/doi/10.1136/bmjqs-2021-013672},\n\tdoi = {10.1136/bmjqs-2021-013672},\n\tabstract = {Background\n Patients and families are important contributors to the diagnostic team, but their perspectives are not reflected in current diagnostic measures. Patients/families can identify some breakdowns in the diagnostic process beyond the clinician’s view. We aimed to develop a framework with patients/families to help organisations identify and categorise patient-reported diagnostic process-related breakdowns (PRDBs) to inform organisational learning.\n \n \n Method\n A multi-stakeholder advisory group including patients, families, clinicians, and experts in diagnostic error, patient engagement and safety, and user-centred design, co-developed a framework for PRDBs in ambulatory care. We tested the framework using standard qualitative analysis methods with two physicians and one patient coder, analysing 2165 patient-reported ambulatory errors in two large surveys representing 25 425 US respondents. We tested intercoder reliability of breakdown categorisation using the Gwet’s AC1 and Cohen’s kappa statistic. We considered agreement coefficients 0.61–0.8=good agreement and 0.81–1.00=excellent agreement.\n \n \n Results\n The framework describes 7 patient-reported breakdown categories (with 40 subcategories), 19 patient-identified contributing factors and 11 potential patient-reported impacts. Patients identified breakdowns in each step of the diagnostic process, including missing or inaccurate main concerns and symptoms; missing/outdated test results; and communication breakdowns such as not feeling heard or misalignment between patient and provider about symptoms, events, or their significance. The frequency of PRDBs was 6.4\\% in one dataset and 6.9\\% in the other. Intercoder reliability showed good-to-excellent reliability in each dataset: AC1 0.89 (95\\% CI 0.89 to 0.90) to 0.96 (95\\% CI 0.95 to 0.97); kappa 0.64 (95\\% CI 0.62, to 0.66) to 0.85 (95\\% CI 0.83 to 0.88).\n \n \n Conclusions\n The PRDB framework, developed in partnership with patients/families, can help organisations identify and reliably categorise PRDBs, including some that are invisible to clinicians; guide interventions to engage patients and families as diagnostic partners; and inform whole organisational learning.},\n\tlanguage = {en},\n\tnumber = {7},\n\turldate = {2026-08-04},\n\tjournal = {BMJ Quality \\& Safety},\n\tauthor = {Bell, Sigall K and Bourgeois, Fabienne and DesRoches, Catherine M and Dong, Joe and Harcourt, Kendall and Liu, Stephen K and Lowe, Elizabeth and McGaffigan, Patricia and Ngo, Long H and Novack, Sandy A and Ralston, James D and Salmi, Liz and Schrandt, Suz and Sheridan, Sue and Sokol-Hessner, Lauge and Thomas, Glenda and Thomas, Eric J},\n\tmonth = jul,\n\tyear = {2022},\n\tpages = {526--540},\n}\n\n\n\n","author_short":["Bell, S. K","Bourgeois, F.","DesRoches, C. M","Dong, J.","Harcourt, K.","Liu, S. K","Lowe, E.","McGaffigan, P.","Ngo, L. H","Novack, S. A","Ralston, J. D","Salmi, L.","Schrandt, S.","Sheridan, S.","Sokol-Hessner, L.","Thomas, G.","Thomas, E. J"],"key":"bell_filling_2022","id":"bell_filling_2022","bibbaseid":"bell-bourgeois-desroches-dong-harcourt-liu-lowe-mcgaffigan-etal-fillingagapinsafetymetricsdevelopmentofapatientcentredframeworktoidentifyandcategorisepatientreportedbreakdownsrelatedtothediagnosticprocessinambulatorycare-2022","role":"author","urls":{"Paper":"https://qualitysafety.bmj.com/lookup/doi/10.1136/bmjqs-2021-013672"},"metadata":{"authorlinks":{}}},"bibtype":"article","biburl":"https://bibbase.org/zotero-group/lizsalmi/6614837","dataSources":["RyfXFmXNStRFjPdoe","tXdjan2EsRaZnLHkJ"],"keywords":[],"search_terms":["filling","gap","safety","metrics","development","patient","centred","framework","identify","categorise","patient","reported","breakdowns","related","diagnostic","process","ambulatory","care","bell","bourgeois","desroches","dong","harcourt","liu","lowe","mcgaffigan","ngo","novack","ralston","salmi","schrandt","sheridan","sokol-hessner","thomas","thomas"],"title":"Filling a gap in safety metrics: development of a patient-centred framework to identify and categorise patient-reported breakdowns related to the diagnostic process in ambulatory care","year":2022}