Comparative Recommender System Evaluation: Benchmarking Recommendation Frameworks

Comparative Recommender System Evaluation: Benchmarking Recommendation Frameworks. Said, A. & Bellogin, A. In RecSys '14, pages 129–136, New York, NY, USA, October, 2014. ACM Press. Journal Abbreviation: RecSys '14

Paper doi abstract bibtex

Recommender systems research is often based on comparisons of predictive accuracy: the better the evaluation scores, the better the recommender. However, it is difficult to compare results from different recommender systems due to the many options in design and implementation of an evaluation strategy. Additionally, algorithmic implementations can diverge from the standard formulation due to manual tuning and modifications that work better in some situations. In this work we compare common recommendation algorithms as implemented in three popular recommendation frameworks. To provide a fair comparison, we have complete control of the evaluation dimensions being benchmarked: dataset, data splitting, evaluation strategies, and metrics. We also include results using the internal evaluation mechanisms of these frameworks. Our analysis points to large differences in recommendation accuracy across frameworks and strategies, i.e. the same baselines may perform orders of magnitude better or worse across frameworks. Our results show the necessity of clear guidelines when reporting evaluation of recommender systems to ensure reproducibility and comparison of results.

@inproceedings{said_comparative_2014,
	address = {New York, NY, USA},
	title = {Comparative {Recommender} {System} {Evaluation}: {Benchmarking} {Recommendation} {Frameworks}},
	url = {http://dx.doi.org/10.1145/2645710.2645746},
	doi = {10.1145/2645710.2645746},
	abstract = {Recommender systems research is often based on comparisons of predictive
accuracy: the better the evaluation scores, the better the recommender.
However, it is difficult to compare results from different recommender
systems due to the many options in design and implementation of an
evaluation strategy. Additionally, algorithmic implementations can diverge
from the standard formulation due to manual tuning and modifications that
work better in some situations. In this work we compare common
recommendation algorithms as implemented in three popular recommendation
frameworks. To provide a fair comparison, we have complete control of the
evaluation dimensions being benchmarked: dataset, data splitting,
evaluation strategies, and metrics. We also include results using the
internal evaluation mechanisms of these frameworks. Our analysis points to
large differences in recommendation accuracy across frameworks and
strategies, i.e. the same baselines may perform orders of magnitude better
or worse across frameworks. Our results show the necessity of clear
guidelines when reporting evaluation of recommender systems to ensure
reproducibility and comparison of results.},
	urldate = {2017-02-03},
	booktitle = {{RecSys} '14},
	publisher = {ACM Press},
	author = {Said, Alan and Bellogin, Alejandro},
	month = oct,
	year = {2014},
	note = {Journal Abbreviation: RecSys '14},
	keywords = {toolkit},
	pages = {129--136},
}

Downloads: 0

{"_id":"4zSwFjqTE6yE3nFPu","bibbaseid":"said-bellogin-comparativerecommendersystemevaluationbenchmarkingrecommendationframeworks-2014","authorIDs":[],"author_short":["Said, A.","Bellogin, A."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","address":"New York, NY, USA","title":"Comparative Recommender System Evaluation: Benchmarking Recommendation Frameworks","url":"http://dx.doi.org/10.1145/2645710.2645746","doi":"10.1145/2645710.2645746","abstract":"Recommender systems research is often based on comparisons of predictive accuracy: the better the evaluation scores, the better the recommender. However, it is difficult to compare results from different recommender systems due to the many options in design and implementation of an evaluation strategy. Additionally, algorithmic implementations can diverge from the standard formulation due to manual tuning and modifications that work better in some situations. In this work we compare common recommendation algorithms as implemented in three popular recommendation frameworks. To provide a fair comparison, we have complete control of the evaluation dimensions being benchmarked: dataset, data splitting, evaluation strategies, and metrics. We also include results using the internal evaluation mechanisms of these frameworks. Our analysis points to large differences in recommendation accuracy across frameworks and strategies, i.e. the same baselines may perform orders of magnitude better or worse across frameworks. Our results show the necessity of clear guidelines when reporting evaluation of recommender systems to ensure reproducibility and comparison of results.","urldate":"2017-02-03","booktitle":"RecSys '14","publisher":"ACM Press","author":[{"propositions":[],"lastnames":["Said"],"firstnames":["Alan"],"suffixes":[]},{"propositions":[],"lastnames":["Bellogin"],"firstnames":["Alejandro"],"suffixes":[]}],"month":"October","year":"2014","note":"Journal Abbreviation: RecSys '14","keywords":"toolkit","pages":"129–136","bibtex":"@inproceedings{said_comparative_2014,\n\taddress = {New York, NY, USA},\n\ttitle = {Comparative {Recommender} {System} {Evaluation}: {Benchmarking} {Recommendation} {Frameworks}},\n\turl = {http://dx.doi.org/10.1145/2645710.2645746},\n\tdoi = {10.1145/2645710.2645746},\n\tabstract = {Recommender systems research is often based on comparisons of predictive\naccuracy: the better the evaluation scores, the better the recommender.\nHowever, it is difficult to compare results from different recommender\nsystems due to the many options in design and implementation of an\nevaluation strategy. Additionally, algorithmic implementations can diverge\nfrom the standard formulation due to manual tuning and modifications that\nwork better in some situations. In this work we compare common\nrecommendation algorithms as implemented in three popular recommendation\nframeworks. To provide a fair comparison, we have complete control of the\nevaluation dimensions being benchmarked: dataset, data splitting,\nevaluation strategies, and metrics. We also include results using the\ninternal evaluation mechanisms of these frameworks. Our analysis points to\nlarge differences in recommendation accuracy across frameworks and\nstrategies, i.e. the same baselines may perform orders of magnitude better\nor worse across frameworks. Our results show the necessity of clear\nguidelines when reporting evaluation of recommender systems to ensure\nreproducibility and comparison of results.},\n\turldate = {2017-02-03},\n\tbooktitle = {{RecSys} '14},\n\tpublisher = {ACM Press},\n\tauthor = {Said, Alan and Bellogin, Alejandro},\n\tmonth = oct,\n\tyear = {2014},\n\tnote = {Journal Abbreviation: RecSys '14},\n\tkeywords = {toolkit},\n\tpages = {129--136},\n}\n\n","author_short":["Said, A.","Bellogin, A."],"key":"said_comparative_2014","id":"said_comparative_2014","bibbaseid":"said-bellogin-comparativerecommendersystemevaluationbenchmarkingrecommendationframeworks-2014","role":"author","urls":{"Paper":"http://dx.doi.org/10.1145/2645710.2645746"},"keyword":["toolkit"],"metadata":{"authorlinks":{}},"downloads":0,"html":""},"bibtype":"inproceedings","biburl":"https://api.zotero.org/users/6655/collections/TJPPJ92X/items?key=VFvZhZXIoHNBbzoLZ1IM2zgf&format=bibtex&limit=100","creationDate":"2020-03-27T02:34:35.372Z","downloads":0,"keywords":["toolkit"],"search_terms":["comparative","recommender","system","evaluation","benchmarking","recommendation","frameworks","said","bellogin"],"title":"Comparative Recommender System Evaluation: Benchmarking Recommendation Frameworks","year":2014,"dataSources":["5Dp4QphkvpvNA33zi","jfoasiDDpStqkkoZB","BiuuFc45aHCgJqDLY"]}