Multiple testing for IR and recommendation system experiments

Multiple testing for IR and recommendation system experiments. Ihemelandu, N. & Ekstrand, M. D. In Proceedings of the 46th European Conference on Information Retrieval, volume 14610, of LNCS, pages 449–457, March, 2024. Springer.

Paper doi abstract bibtex

While there has been significant research on statistical techniques for comparing two information retrieval (IR) systems, many IR experiments test more than two systems. This can lead to inflated false discoveries due to the multiple-comparison problem (MCP). A few IR studies have investigated multiple comparison procedures; these studies mostly use TREC data and control the familywise error rate. In this study, we extend their investigation to include recommendation system evaluation data as well as multiple comparison procedures that controls for False Discovery Rate (FDR).

@inproceedings{ihemelandu_multiple_2024,
	series = {{LNCS}},
	title = {Multiple testing for {IR} and recommendation system experiments},
	volume = {14610},
	url = {https://md.ekstrandom.net/pubs/ecir-mcp},
	doi = {10.1007/978-3-031-56063-7_37},
	abstract = {While there has been significant research on statistical techniques for comparing two information retrieval (IR) systems, many IR experiments test more than two systems. This can lead to inflated false discoveries due to the multiple-comparison problem (MCP). A few IR studies have investigated multiple comparison procedures; these studies mostly use TREC data and control the familywise error rate. In this study, we extend their investigation to include recommendation system evaluation data as well as multiple comparison procedures that controls for False Discovery Rate (FDR).},
	language = {en},
	urldate = {2024-01-04},
	booktitle = {Proceedings of the 46th {European} {Conference} on {Information} {Retrieval}},
	publisher = {Springer},
	author = {Ihemelandu, Ngozi and Ekstrand, Michael D.},
	month = mar,
	year = {2024},
	pages = {449--457},
}

Downloads: 0

{"_id":"BNRzPgatCFw3czdk2","bibbaseid":"ihemelandu-ekstrand-multipletestingforirandrecommendationsystemexperiments-2024","author_short":["Ihemelandu, N.","Ekstrand, M. D."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","series":"LNCS","title":"Multiple testing for IR and recommendation system experiments","volume":"14610","url":"https://md.ekstrandom.net/pubs/ecir-mcp","doi":"10.1007/978-3-031-56063-7_37","abstract":"While there has been significant research on statistical techniques for comparing two information retrieval (IR) systems, many IR experiments test more than two systems. This can lead to inflated false discoveries due to the multiple-comparison problem (MCP). A few IR studies have investigated multiple comparison procedures; these studies mostly use TREC data and control the familywise error rate. In this study, we extend their investigation to include recommendation system evaluation data as well as multiple comparison procedures that controls for False Discovery Rate (FDR).","language":"en","urldate":"2024-01-04","booktitle":"Proceedings of the 46th European Conference on Information Retrieval","publisher":"Springer","author":[{"propositions":[],"lastnames":["Ihemelandu"],"firstnames":["Ngozi"],"suffixes":[]},{"propositions":[],"lastnames":["Ekstrand"],"firstnames":["Michael","D."],"suffixes":[]}],"month":"March","year":"2024","pages":"449–457","bibtex":"@inproceedings{ihemelandu_multiple_2024,\n\tseries = {{LNCS}},\n\ttitle = {Multiple testing for {IR} and recommendation system experiments},\n\tvolume = {14610},\n\turl = {https://md.ekstrandom.net/pubs/ecir-mcp},\n\tdoi = {10.1007/978-3-031-56063-7_37},\n\tabstract = {While there has been significant research on statistical techniques for comparing two information retrieval (IR) systems, many IR experiments test more than two systems. This can lead to inflated false discoveries due to the multiple-comparison problem (MCP). A few IR studies have investigated multiple comparison procedures; these studies mostly use TREC data and control the familywise error rate. In this study, we extend their investigation to include recommendation system evaluation data as well as multiple comparison procedures that controls for False Discovery Rate (FDR).},\n\tlanguage = {en},\n\turldate = {2024-01-04},\n\tbooktitle = {Proceedings of the 46th {European} {Conference} on {Information} {Retrieval}},\n\tpublisher = {Springer},\n\tauthor = {Ihemelandu, Ngozi and Ekstrand, Michael D.},\n\tmonth = mar,\n\tyear = {2024},\n\tpages = {449--457},\n}\n\n","author_short":["Ihemelandu, N.","Ekstrand, M. D."],"key":"ihemelandu_multiple_2024","id":"ihemelandu_multiple_2024","bibbaseid":"ihemelandu-ekstrand-multipletestingforirandrecommendationsystemexperiments-2024","role":"author","urls":{"Paper":"https://md.ekstrandom.net/pubs/ecir-mcp"},"metadata":{"authorlinks":{}}},"bibtype":"inproceedings","biburl":"https://api.zotero.org/users/6655/collections/3TB3KT36/items?key=VFvZhZXIoHNBbzoLZ1IM2zgf&format=bibtex&limit=100","dataSources":["7KNAjxiv2tsagmbgY"],"keywords":[],"search_terms":["multiple","testing","recommendation","system","experiments","ihemelandu","ekstrand"],"title":"Multiple testing for IR and recommendation system experiments","year":2024}