User perception of differences in movie recommendation algorithms. Ekstrand, M. D, Harper, F M., Willemsen, M. C, & Konstan, J. A In Proceedings of the Eighth ACM Conference on Recommender Systems, pages 161–168, New York, NY, USA, October, 2014. ACM. Journal Abbreviation: RecSys '14
Paper doi abstract bibtex Recommender systems research is often based on comparisons of predictive accuracy: the better the evaluation scores, the better the recommender. However, it is difficult to compare results from different recommender systems due to the many options in design and implementation of an evaluation strategy. Additionally, algorithm implementations can diverge from the standard formulation due to manual tuning and modifications that work better in some situations. In this work we compare common recommendation algorithms as implemented in three popular recommendation frameworks. To provide a fair comparison, we have complete control of the evaluation dimensions being benchmarked: dataset, data splitting, evaluation strategies, and metrics. We also include results using the internal evaluation mechanisms of these frameworks. Our analysis points to large differences in recommendation accuracy across frameworks and strategies, i.e. the same baselines may perform orders of magnitude better or worse across frameworks. Our results show the necessity of clear guidelines when reporting evaluation of recommender systems to ensure reproducibility and comparison of results.
@inproceedings{ekstrand_user_2014,
address = {New York, NY, USA},
title = {User perception of differences in movie recommendation algorithms},
url = {http://dx.doi.org/10.1145/2645710.2645737},
doi = {10.1145/2645710.2645737},
abstract = {Recommender systems research is often based on comparisons of predictive
accuracy: the better the evaluation scores, the better the recommender.
However, it is difficult to compare results from different recommender
systems due to the many options in design and implementation of an
evaluation strategy. Additionally, algorithm implementations can diverge
from the standard formulation due to manual tuning and modifications that
work better in some situations. In this work we compare common
recommendation algorithms as implemented in three popular recommendation
frameworks. To provide a fair comparison, we have complete control of the
evaluation dimensions being benchmarked: dataset, data splitting,
evaluation strategies, and metrics. We also include results using the
internal evaluation mechanisms of these frameworks. Our analysis points to
large differences in recommendation accuracy across frameworks and
strategies, i.e. the same baselines may perform orders of magnitude better
or worse across frameworks. Our results show the necessity of clear
guidelines when reporting evaluation of recommender systems to ensure
reproducibility and comparison of results.},
booktitle = {Proceedings of the {Eighth} {ACM} {Conference} on {Recommender} {Systems}},
publisher = {ACM},
author = {Ekstrand, Michael D and Harper, F Maxwell and Willemsen, Martijn C and Konstan, Joseph A},
month = oct,
year = {2014},
note = {Journal Abbreviation: RecSys '14},
pages = {161--168},
}
Downloads: 0
{"_id":"pPcqJLbNKJYQMtiqC","bibbaseid":"ekstrand-harper-willemsen-konstan-userperceptionofdifferencesinmovierecommendationalgorithms-2014","authorIDs":[],"author_short":["Ekstrand, M. D","Harper, F M.","Willemsen, M. C","Konstan, J. A"],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","address":"New York, NY, USA","title":"User perception of differences in movie recommendation algorithms","url":"http://dx.doi.org/10.1145/2645710.2645737","doi":"10.1145/2645710.2645737","abstract":"Recommender systems research is often based on comparisons of predictive accuracy: the better the evaluation scores, the better the recommender. However, it is difficult to compare results from different recommender systems due to the many options in design and implementation of an evaluation strategy. Additionally, algorithm implementations can diverge from the standard formulation due to manual tuning and modifications that work better in some situations. In this work we compare common recommendation algorithms as implemented in three popular recommendation frameworks. To provide a fair comparison, we have complete control of the evaluation dimensions being benchmarked: dataset, data splitting, evaluation strategies, and metrics. We also include results using the internal evaluation mechanisms of these frameworks. Our analysis points to large differences in recommendation accuracy across frameworks and strategies, i.e. the same baselines may perform orders of magnitude better or worse across frameworks. Our results show the necessity of clear guidelines when reporting evaluation of recommender systems to ensure reproducibility and comparison of results.","booktitle":"Proceedings of the Eighth ACM Conference on Recommender Systems","publisher":"ACM","author":[{"propositions":[],"lastnames":["Ekstrand"],"firstnames":["Michael","D"],"suffixes":[]},{"propositions":[],"lastnames":["Harper"],"firstnames":["F","Maxwell"],"suffixes":[]},{"propositions":[],"lastnames":["Willemsen"],"firstnames":["Martijn","C"],"suffixes":[]},{"propositions":[],"lastnames":["Konstan"],"firstnames":["Joseph","A"],"suffixes":[]}],"month":"October","year":"2014","note":"Journal Abbreviation: RecSys '14","pages":"161–168","bibtex":"@inproceedings{ekstrand_user_2014,\n\taddress = {New York, NY, USA},\n\ttitle = {User perception of differences in movie recommendation algorithms},\n\turl = {http://dx.doi.org/10.1145/2645710.2645737},\n\tdoi = {10.1145/2645710.2645737},\n\tabstract = {Recommender systems research is often based on comparisons of predictive\naccuracy: the better the evaluation scores, the better the recommender.\nHowever, it is difficult to compare results from different recommender\nsystems due to the many options in design and implementation of an\nevaluation strategy. Additionally, algorithm implementations can diverge\nfrom the standard formulation due to manual tuning and modifications that\nwork better in some situations. In this work we compare common\nrecommendation algorithms as implemented in three popular recommendation\nframeworks. To provide a fair comparison, we have complete control of the\nevaluation dimensions being benchmarked: dataset, data splitting,\nevaluation strategies, and metrics. We also include results using the\ninternal evaluation mechanisms of these frameworks. Our analysis points to\nlarge differences in recommendation accuracy across frameworks and\nstrategies, i.e. the same baselines may perform orders of magnitude better\nor worse across frameworks. Our results show the necessity of clear\nguidelines when reporting evaluation of recommender systems to ensure\nreproducibility and comparison of results.},\n\tbooktitle = {Proceedings of the {Eighth} {ACM} {Conference} on {Recommender} {Systems}},\n\tpublisher = {ACM},\n\tauthor = {Ekstrand, Michael D and Harper, F Maxwell and Willemsen, Martijn C and Konstan, Joseph A},\n\tmonth = oct,\n\tyear = {2014},\n\tnote = {Journal Abbreviation: RecSys '14},\n\tpages = {161--168},\n}\n\n","author_short":["Ekstrand, M. D","Harper, F M.","Willemsen, M. C","Konstan, J. A"],"key":"ekstrand_user_2014","id":"ekstrand_user_2014","bibbaseid":"ekstrand-harper-willemsen-konstan-userperceptionofdifferencesinmovierecommendationalgorithms-2014","role":"author","urls":{"Paper":"http://dx.doi.org/10.1145/2645710.2645737"},"metadata":{"authorlinks":{}},"downloads":0},"bibtype":"inproceedings","biburl":"https://api.zotero.org/users/6655/collections/TJPPJ92X/items?key=VFvZhZXIoHNBbzoLZ1IM2zgf&format=bibtex&limit=100","creationDate":"2020-03-27T02:34:35.369Z","downloads":0,"keywords":[],"search_terms":["user","perception","differences","movie","recommendation","algorithms","ekstrand","harper","willemsen","konstan"],"title":"User perception of differences in movie recommendation algorithms","year":2014,"dataSources":["5Dp4QphkvpvNA33zi","jfoasiDDpStqkkoZB","BiuuFc45aHCgJqDLY"]}