CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias. Gupta, V., Venkit, P. N., Laurencon, H., Wilson, S., & Passonneau, R. J. In Proceedings of the 1st Conference on Language Modeling (COLM), Philadelphia, PA, July, 2024. OpenReview.
CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias [link]Paper  abstract   bibtex   
As language models (LMs) become increasingly powerful and widely used, it is important to quantify them for sociodemographic bias with potential for harm. Prior measures of bias are sensitive to perturbations in the templates designed to compare performance across social groups, due to factors such as low diversity or limited number of templates. Also, most previous work considers only one NLP task. We introduce Comprehensive Assessment of Language Models (CALM) for robust measurement of social biases. We use sixteen datasets for question-answering, sentiment analysis and natural language inference and filter them to produce 224 templates with high diversity (e.g., length, vocabulary). This helps us create a novel dataset of 78,400 prompts covering the three NLP tasks. Our empirical evaluation shows that CALM bias scores are more robust and far less sensitive than previous bias measurements to perturbations in the templates, such as synonym substitution, or to random subset selection of templates. We apply CALM to 20 large language models, and find that for 2 LM series, larger parameter models tend to be more biased than smaller ones. The T0 series is the least biased model families, of the 20 LLMs investigated here.
@inproceedings{gupta_calm_2024,
	address = {Philadelphia, PA},
	title = {{CALM} : {A} {Multi}-task {Benchmark} for {Comprehensive} {Assessment} of {Language} {Model} {Bias}},
	url = {https://openreview.net/forum?id=RLFca3arx7},
	abstract = {As language models (LMs) become increasingly powerful and widely used, it is important to quantify them for sociodemographic bias with potential for harm. Prior measures of bias are sensitive to perturbations in the templates designed to compare performance across social groups, due to factors such as low diversity or limited number of templates. Also, most previous work considers only one NLP task. We introduce Comprehensive Assessment of Language Models (CALM) for robust measurement of social biases. We use sixteen datasets for question-answering, sentiment analysis and natural language inference and filter them to produce 224 templates with high diversity (e.g., length, vocabulary). This helps us create a novel dataset of 78,400 prompts covering the three NLP tasks. Our empirical evaluation shows that CALM bias scores are more robust and far less sensitive than previous bias measurements to perturbations in the templates, such as synonym substitution, or to random subset selection of templates. We apply CALM to 20 large language models, and find that for 2 LM series, larger parameter models tend to be more biased than smaller ones. The T0 series is the least biased model families, of the 20 LLMs investigated here.},
	booktitle = {Proceedings of the 1st {Conference} on {Language} {Modeling} ({COLM})},
	publisher = {OpenReview},
	author = {Gupta, Vipul and Venkit, Pranav Narayanan and Laurencon, Hugo and Wilson, Shomir and Passonneau, Rebecca J.},
	month = jul,
	year = {2024},
}
Downloads: 0
About
Documentation
Keywords
Search
Users
Terms and Conditions of Use
Privacy Policy
Sitemap
© BibBase.org 2021