I-trustworthy Models. A framework for trustworthiness evaluation of probabilistic classifiers. Vashistha, R. & Farahi, A. In PMLR, volume 258, Mai Khao, Thailand, 2025. abstract bibtex As probabilistic models continue to permeate various facets of our society and contribute to scientific advancements, it becomes a necessity to go beyond traditional metrics such as predictive accuracy and error rates and assess their trustworthiness. Grounded in the competence-based theory of trust, this work formalizes I-trustworthy framework – a novel framework for assessing the trustworthiness of probabilistic classifiers for inference tasks by linking local calibration to trustworthiness. To assess I-trustworthiness, we use the local calibration error (LCE) and develop a method of hypothesis-testing. This method utilizes a kernel-based test statistic, Kernel Local Calibration Error (KLCE), to test local calibration of a probabilistic classifier. This study provides theoretical guarantees by o!ering convergence bounds for an unbiased estimator of KLCE. Additionally, we present a diagnostic tool designed to identify and measure biases in cases of miscalibration. The e!ectiveness of the proposed test statistic is demonstrated through its application to both simulated and real-world datasets. Finally, LCE of related recalibration methods is studied, and we provide evidence of insu"ciency of existing methods to achieve I-trustworthiness.
@inproceedings{vashistha_i-trustworthy_2025,
address = {Mai Khao, Thailand},
title = {I-trustworthy {Models}. {A} framework for trustworthiness evaluation of probabilistic classifiers},
volume = {258},
abstract = {As probabilistic models continue to permeate various facets of our society and contribute to scientific advancements, it becomes a necessity to go beyond traditional metrics such as predictive accuracy and error rates and assess their trustworthiness. Grounded in the competence-based theory of trust, this work formalizes I-trustworthy framework – a novel framework for assessing the trustworthiness of probabilistic classifiers for inference tasks by linking local calibration to trustworthiness. To assess I-trustworthiness, we use the local calibration error (LCE) and develop a method of hypothesis-testing. This method utilizes a kernel-based test statistic, Kernel Local Calibration Error (KLCE), to test local calibration of a probabilistic classifier. This study provides theoretical guarantees by o!ering convergence bounds for an unbiased estimator of KLCE. Additionally, we present a diagnostic tool designed to identify and measure biases in cases of miscalibration. The e!ectiveness of the proposed test statistic is demonstrated through its application to both simulated and real-world datasets. Finally, LCE of related recalibration methods is studied, and we provide evidence of insu"ciency of existing methods to achieve I-trustworthiness.},
language = {en},
booktitle = {{PMLR}},
author = {Vashistha, Ritwik and Farahi, Arya},
year = {2025},
keywords = {Explainable},
}
Downloads: 0
{"_id":"uaf9nJGA6LuEwYFAj","bibbaseid":"vashistha-farahi-itrustworthymodelsaframeworkfortrustworthinessevaluationofprobabilisticclassiers-2025","author_short":["Vashistha, R.","Farahi, A."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","address":"Mai Khao, Thailand","title":"I-trustworthy Models. A framework for trustworthiness evaluation of probabilistic classifiers","volume":"258","abstract":"As probabilistic models continue to permeate various facets of our society and contribute to scientific advancements, it becomes a necessity to go beyond traditional metrics such as predictive accuracy and error rates and assess their trustworthiness. Grounded in the competence-based theory of trust, this work formalizes I-trustworthy framework – a novel framework for assessing the trustworthiness of probabilistic classifiers for inference tasks by linking local calibration to trustworthiness. To assess I-trustworthiness, we use the local calibration error (LCE) and develop a method of hypothesis-testing. This method utilizes a kernel-based test statistic, Kernel Local Calibration Error (KLCE), to test local calibration of a probabilistic classifier. This study provides theoretical guarantees by o!ering convergence bounds for an unbiased estimator of KLCE. Additionally, we present a diagnostic tool designed to identify and measure biases in cases of miscalibration. The e!ectiveness of the proposed test statistic is demonstrated through its application to both simulated and real-world datasets. Finally, LCE of related recalibration methods is studied, and we provide evidence of insu\"ciency of existing methods to achieve I-trustworthiness.","language":"en","booktitle":"PMLR","author":[{"propositions":[],"lastnames":["Vashistha"],"firstnames":["Ritwik"],"suffixes":[]},{"propositions":[],"lastnames":["Farahi"],"firstnames":["Arya"],"suffixes":[]}],"year":"2025","keywords":"Explainable","bibtex":"@inproceedings{vashistha_i-trustworthy_2025,\n\taddress = {Mai Khao, Thailand},\n\ttitle = {I-trustworthy {Models}. {A} framework for trustworthiness evaluation of probabilistic classifiers},\n\tvolume = {258},\n\tabstract = {As probabilistic models continue to permeate various facets of our society and contribute to scientific advancements, it becomes a necessity to go beyond traditional metrics such as predictive accuracy and error rates and assess their trustworthiness. Grounded in the competence-based theory of trust, this work formalizes I-trustworthy framework – a novel framework for assessing the trustworthiness of probabilistic classifiers for inference tasks by linking local calibration to trustworthiness. To assess I-trustworthiness, we use the local calibration error (LCE) and develop a method of hypothesis-testing. This method utilizes a kernel-based test statistic, Kernel Local Calibration Error (KLCE), to test local calibration of a probabilistic classifier. This study provides theoretical guarantees by o!ering convergence bounds for an unbiased estimator of KLCE. Additionally, we present a diagnostic tool designed to identify and measure biases in cases of miscalibration. The e!ectiveness of the proposed test statistic is demonstrated through its application to both simulated and real-world datasets. Finally, LCE of related recalibration methods is studied, and we provide evidence of insu\"ciency of existing methods to achieve I-trustworthiness.},\n\tlanguage = {en},\n\tbooktitle = {{PMLR}},\n\tauthor = {Vashistha, Ritwik and Farahi, Arya},\n\tyear = {2025},\n\tkeywords = {Explainable},\n}\n\n\n\n","author_short":["Vashistha, R.","Farahi, A."],"key":"vashistha_i-trustworthy_2025","id":"vashistha_i-trustworthy_2025","bibbaseid":"vashistha-farahi-itrustworthymodelsaframeworkfortrustworthinessevaluationofprobabilisticclassiers-2025","role":"author","urls":{},"keyword":["Explainable"],"metadata":{"authorlinks":{}}},"bibtype":"inproceedings","biburl":"https://bibbase.org/zotero-group/pratikmhatre/5933976","dataSources":["yJr5AAtJ5Sz3Q4WT4"],"keywords":["explainable"],"search_terms":["trustworthy","models","framework","trustworthiness","evaluation","probabilistic","classi","ers","vashistha","farahi"],"title":"I-trustworthy Models. A framework for trustworthiness evaluation of probabilistic classifiers","year":2025}