Sturgeon and the Cool Kids: Problems with Top-N Recommender Evaluation

Sturgeon and the Cool Kids: Problems with Top-N Recommender Evaluation. Ekstrand, M. D & Mahant, V. In Proceedings of the 30th Florida Artificial Intelligence Research Society Conference, of FLAIRS 30, May, 2017. AAAI Press.

Paper abstract bibtex

Top-N evaluation of recommender systems, typically carried out using metrics from information retrieval or machine learning, has several challenges. Two of these challenges are popularity bias, where the evaluation intrinsically favors algorithms that recommend popular items, and misclassified decoys, where items for which no user relevance is known are actually relevant to the user, but the evaluation is unaware and penalizes the recommender for suggesting them. One strategy for mitigating the misclassified decoy problem is the one-plus-random evaluation strategy and its generalization, which we call random decoys. In this work, we explore the random decoy strategy through both a theoretical treatment and an empirical study, but find little evidence to guide its tuning and show that it has complex and deleterious interactions with popularity bias.

@inproceedings{ekstrand_sturgeon_2017,
	series = {{FLAIRS} 30},
	title = {Sturgeon and the {Cool} {Kids}: {Problems} with {Top}-{N} {Recommender} {Evaluation}},
	url = {https://aaai.org/papers/639-flairs-2017-15534/},
	abstract = {Top-N evaluation of recommender systems, typically carried out using
metrics from information retrieval or machine learning, has several
challenges. Two of these challenges are popularity bias, where the
evaluation intrinsically favors algorithms that recommend popular items,
and misclassified decoys, where items for which no user relevance is known
are actually relevant to the user, but the evaluation is unaware and
penalizes the recommender for suggesting them. One strategy for mitigating
the misclassified decoy problem is the one-plus-random evaluation strategy
and its generalization, which we call random decoys. In this work, we
explore the random decoy strategy through both a theoretical treatment and
an empirical study, but find little evidence to guide its tuning and show
that it has complex and deleterious interactions with popularity bias.},
	booktitle = {Proceedings of the 30th {Florida} {Artificial} {Intelligence} {Research} {Society} {Conference}},
	publisher = {AAAI Press},
	author = {Ekstrand, Michael D and Mahant, Vaibhav},
	month = may,
	year = {2017},
}

Downloads: 0

{"_id":"2WEP3mHoMPtwZMFwM","bibbaseid":"ekstrand-mahant-sturgeonandthecoolkidsproblemswithtopnrecommenderevaluation-2017","authorIDs":[],"author_short":["Ekstrand, M. D","Mahant, V."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","series":"FLAIRS 30","title":"Sturgeon and the Cool Kids: Problems with Top-N Recommender Evaluation","url":"https://aaai.org/papers/639-flairs-2017-15534/","abstract":"Top-N evaluation of recommender systems, typically carried out using metrics from information retrieval or machine learning, has several challenges. Two of these challenges are popularity bias, where the evaluation intrinsically favors algorithms that recommend popular items, and misclassified decoys, where items for which no user relevance is known are actually relevant to the user, but the evaluation is unaware and penalizes the recommender for suggesting them. One strategy for mitigating the misclassified decoy problem is the one-plus-random evaluation strategy and its generalization, which we call random decoys. In this work, we explore the random decoy strategy through both a theoretical treatment and an empirical study, but find little evidence to guide its tuning and show that it has complex and deleterious interactions with popularity bias.","booktitle":"Proceedings of the 30th Florida Artificial Intelligence Research Society Conference","publisher":"AAAI Press","author":[{"propositions":[],"lastnames":["Ekstrand"],"firstnames":["Michael","D"],"suffixes":[]},{"propositions":[],"lastnames":["Mahant"],"firstnames":["Vaibhav"],"suffixes":[]}],"month":"May","year":"2017","bibtex":"@inproceedings{ekstrand_sturgeon_2017,\n\tseries = {{FLAIRS} 30},\n\ttitle = {Sturgeon and the {Cool} {Kids}: {Problems} with {Top}-{N} {Recommender} {Evaluation}},\n\turl = {https://aaai.org/papers/639-flairs-2017-15534/},\n\tabstract = {Top-N evaluation of recommender systems, typically carried out using\nmetrics from information retrieval or machine learning, has several\nchallenges. Two of these challenges are popularity bias, where the\nevaluation intrinsically favors algorithms that recommend popular items,\nand misclassified decoys, where items for which no user relevance is known\nare actually relevant to the user, but the evaluation is unaware and\npenalizes the recommender for suggesting them. One strategy for mitigating\nthe misclassified decoy problem is the one-plus-random evaluation strategy\nand its generalization, which we call random decoys. In this work, we\nexplore the random decoy strategy through both a theoretical treatment and\nan empirical study, but find little evidence to guide its tuning and show\nthat it has complex and deleterious interactions with popularity bias.},\n\tbooktitle = {Proceedings of the 30th {Florida} {Artificial} {Intelligence} {Research} {Society} {Conference}},\n\tpublisher = {AAAI Press},\n\tauthor = {Ekstrand, Michael D and Mahant, Vaibhav},\n\tmonth = may,\n\tyear = {2017},\n}\n\n","author_short":["Ekstrand, M. D","Mahant, V."],"key":"ekstrand_sturgeon_2017","id":"ekstrand_sturgeon_2017","bibbaseid":"ekstrand-mahant-sturgeonandthecoolkidsproblemswithtopnrecommenderevaluation-2017","role":"author","urls":{"Paper":"https://aaai.org/papers/639-flairs-2017-15534/"},"metadata":{"authorlinks":{}},"downloads":0},"bibtype":"inproceedings","biburl":"https://api.zotero.org/users/6655/collections/TJPPJ92X/items?key=VFvZhZXIoHNBbzoLZ1IM2zgf&format=bibtex&limit=100","creationDate":"2020-03-27T02:34:35.291Z","downloads":0,"keywords":[],"search_terms":["sturgeon","cool","kids","problems","top","recommender","evaluation","ekstrand","mahant"],"title":"Sturgeon and the Cool Kids: Problems with Top-N Recommender Evaluation","year":2017,"dataSources":["5Dp4QphkvpvNA33zi","jfoasiDDpStqkkoZB","BiuuFc45aHCgJqDLY"]}