All the cool kids, how do they fit in?: popularity and demographic biases in recommender evaluation and effectiveness. Ekstrand, M. D, Tian, M., Azpiazu, I. M., Ekstrand, J. D, Anuyah, O., McNeill, D., & Pera, M. S. In Friedler, S. A & Wilson, C., editors, Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81, of Proceedings of Machine Learning Research, pages 172–186, 2018. PMLR. Journal Abbreviation: Proceedings of Machine Learning Research
All the cool kids, how do they fit in?: popularity and demographic biases in recommender evaluation and effectiveness [link]Paper  abstract   bibtex   
In the research literature, evaluations of recommender system effectiveness typically report results over a given data set, providing an aggregate measure of effectiveness over each instance (e.g. user) in the data set. Recent advances in information retrieval evaluation, however, demonstrate the importance of considering the distribution of effectiveness across diverse groups of varying sizes. For example, do users of different ages or genders obtain similar utility from the system, particularly if their group is a relatively small subset of the user base? We apply this consideration to recommender systems, using offline evaluation and a utility-based metric of recommendation effectiveness to explore whether different user demographic groups experience similar recommendation accuracy. We find demographic differences in measured recommender effectiveness across two data sets containing different types of feedback in different domains; these differences sometimes, but not always, correlate with the size of the user group in question. Demographic effects also have a complex—and likely detrimental—interaction with popularity bias, a known deficiency of recommender evaluation. These results demonstrate the need for recommender system evaluation protocols that explicitly quantify the degree to which the system is meeting the information needs of all its users, as well as the need for researchers and operators to move beyond naïve evaluations that favor the needs of larger subsets of the user population while ignoring smaller subsets.
@inproceedings{ekstrand_all_2018,
	series = {Proceedings of {Machine} {Learning} {Research}},
	title = {All the cool kids, how do they fit in?: popularity and demographic biases in recommender evaluation and effectiveness},
	volume = {81},
	url = {https://proceedings.mlr.press/v81/ekstrand18b.html},
	abstract = {In the research literature, evaluations of recommender system
effectiveness typically report results over a given data set, providing an
aggregate measure of effectiveness over each instance (e.g. user) in the
data set. Recent advances in information retrieval evaluation, however,
demonstrate the importance of considering the distribution of
effectiveness across diverse groups of varying sizes. For example, do
users of different ages or genders obtain similar utility from the system,
particularly if their group is a relatively small subset of the user base?
We apply this consideration to recommender systems, using offline
evaluation and a utility-based metric of recommendation effectiveness to
explore whether different user demographic groups experience similar
recommendation accuracy. We find demographic differences in measured
recommender effectiveness across two data sets containing different types
of feedback in different domains; these differences sometimes, but not
always, correlate with the size of the user group in question. Demographic
effects also have a complex—and likely detrimental—interaction with
popularity bias, a known deficiency of recommender evaluation. These
results demonstrate the need for recommender system evaluation protocols
that explicitly quantify the degree to which the system is meeting the
information needs of all its users, as well as the need for researchers
and operators to move beyond naïve evaluations that favor the needs of
larger subsets of the user population while ignoring smaller subsets.},
	booktitle = {Proceedings of the 1st {Conference} on {Fairness}, {Accountability} and {Transparency}},
	publisher = {PMLR},
	author = {Ekstrand, Michael D and Tian, Mucun and Azpiazu, Ion Madrazo and Ekstrand, Jennifer D and Anuyah, Oghenemaro and McNeill, David and Pera, Maria Soledad},
	editor = {Friedler, Sorelle A and Wilson, Christo},
	year = {2018},
	note = {Journal Abbreviation: Proceedings of Machine Learning Research},
	pages = {172--186},
}
Downloads: 0
About
Documentation
Keywords
Search
Users
Terms and Conditions of Use
Privacy Policy
Sitemap
© BibBase.org 2021