Leveraging the Contributions of the Casual Majority to Identify AppealingWeb Content. Hogg, T. & Lerman, K. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2016), November, 2016. AAAI. Paper abstract bibtex 2 downloads Users of peer production web sites differ greatly in their activity levels. A small minority are engaged contributors, while the vast majority are only casual surfers. The casual users devote little effort to evaluating the site�s content and many of them visit the site only once. This churn poses a challenge for sites attempting to gauge user interest in their content. The challenge is especially severe for sites focusing on content with subjective quality, including movies, music, restaurants and items in other cultural markets. A key question is whether content evaluation should use opinions of all users or only the minority who devote significant effort to reviewing content? Using Amazon Mechanical Turk, we experimentally address this question by comparing outcomes for these two approaches. We find that the larger numbers of less informed users more than offset their noisy signals on content quality to provide rapid evaluation. However, such users are systematically biased, and the speed of their assessments comes at the expense of limited collective accuracy.
@INPROCEEDINGS{Hogg2016hcomp,
author = {Tad Hogg and Kristina Lerman},
title = {Leveraging the Contributions of the Casual Majority to Identify AppealingWeb
Content},
booktitle = {Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2016)},
year = {2016},
month = {November},
publisher = {AAAI},
keywords = {crowdsourcing},
abstract={Users of peer production web sites differ greatly in their activity levels. A small minority are engaged contributors, while the vast majority are only casual surfers. The casual users devote little effort to evaluating the site�s content and many of them visit the site only once. This churn poses a challenge for sites attempting to gauge user interest in their content. The challenge is especially severe for sites focusing on content with subjective quality, including movies, music, restaurants and items in other cultural markets. A key question is whether content evaluation should use opinions of all users or only the minority who devote significant effort to reviewing content? Using Amazon Mechanical Turk, we experimentally address this question by comparing outcomes for these two approaches. We find that the larger numbers of less informed users more than offset their noisy signals on content quality to provide rapid evaluation. However, such users are systematically biased, and the speed of their assessments comes at the expense of limited collective accuracy.},
url={http://www.isi.edu/integration/people/lerman/papers/hogg-hcomp.pdf}
}
Downloads: 2
{"_id":"aFZQGfToEXoFNbg9H","bibbaseid":"hogg-lerman-leveragingthecontributionsofthecasualmajoritytoidentifyappealingwebcontent-2016","downloads":2,"creationDate":"2016-09-16T15:40:01.702Z","title":"Leveraging the Contributions of the Casual Majority to Identify AppealingWeb Content","author_short":["Hogg, T.","Lerman, K."],"year":2016,"bibtype":"inproceedings","biburl":"https://bibbase.org/network/files/iNQKC4NCiGYaef6D9","bibdata":{"bibtype":"inproceedings","type":"inproceedings","author":[{"firstnames":["Tad"],"propositions":[],"lastnames":["Hogg"],"suffixes":[]},{"firstnames":["Kristina"],"propositions":[],"lastnames":["Lerman"],"suffixes":[]}],"title":"Leveraging the Contributions of the Casual Majority to Identify AppealingWeb Content","booktitle":"Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2016)","year":"2016","month":"November","publisher":"AAAI","keywords":"crowdsourcing","abstract":"Users of peer production web sites differ greatly in their activity levels. A small minority are engaged contributors, while the vast majority are only casual surfers. The casual users devote little effort to evaluating the site�s content and many of them visit the site only once. This churn poses a challenge for sites attempting to gauge user interest in their content. The challenge is especially severe for sites focusing on content with subjective quality, including movies, music, restaurants and items in other cultural markets. A key question is whether content evaluation should use opinions of all users or only the minority who devote significant effort to reviewing content? Using Amazon Mechanical Turk, we experimentally address this question by comparing outcomes for these two approaches. We find that the larger numbers of less informed users more than offset their noisy signals on content quality to provide rapid evaluation. However, such users are systematically biased, and the speed of their assessments comes at the expense of limited collective accuracy.","url":"http://www.isi.edu/integration/people/lerman/papers/hogg-hcomp.pdf","bibtex":"@INPROCEEDINGS{Hogg2016hcomp,\n author = {Tad Hogg and Kristina Lerman},\n title = {Leveraging the Contributions of the Casual Majority to Identify AppealingWeb\nContent},\n booktitle = {Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2016)},\n year = {2016},\n month = {November},\n publisher = {AAAI},\n keywords = {crowdsourcing},\n abstract={Users of peer production web sites differ greatly in their activity levels. A small minority are engaged contributors, while the vast majority are only casual surfers. The casual users devote little effort to evaluating the site�s content and many of them visit the site only once. This churn poses a challenge for sites attempting to gauge user interest in their content. The challenge is especially severe for sites focusing on content with subjective quality, including movies, music, restaurants and items in other cultural markets. A key question is whether content evaluation should use opinions of all users or only the minority who devote significant effort to reviewing content? Using Amazon Mechanical Turk, we experimentally address this question by comparing outcomes for these two approaches. We find that the larger numbers of less informed users more than offset their noisy signals on content quality to provide rapid evaluation. However, such users are systematically biased, and the speed of their assessments comes at the expense of limited collective accuracy.},\n url={http://www.isi.edu/integration/people/lerman/papers/hogg-hcomp.pdf}\n}\n\n","author_short":["Hogg, T.","Lerman, K."],"key":"Hogg2016hcomp","id":"Hogg2016hcomp","bibbaseid":"hogg-lerman-leveragingthecontributionsofthecasualmajoritytoidentifyappealingwebcontent-2016","role":"author","urls":{"Paper":"http://www.isi.edu/integration/people/lerman/papers/hogg-hcomp.pdf"},"keyword":["crowdsourcing"],"metadata":{"authorlinks":{"lerman, k":"https://www.isi.edu/people-lerman/publications/"}},"downloads":2},"search_terms":["leveraging","contributions","casual","majority","identify","appealingweb","content","hogg","lerman"],"keywords":["crowdsourcing"],"authorIDs":["2GLrmqhwwX9DFA6ae","2dqDdyMfgPcMjHi5F","2jfSjPe6myq6vH4Qn","2xvjC3SWWQTRsXyZA","3CAqXRQM2QByQRkGn","3JZKbDS4vm8DjTioB","4Zu7FeDfFpSJG5qwH","4aMdkYHwN8YxtdJnt","545710982abc8e9f3700001d","5de7ebdec8f9f6df010001ca","5deaed9becff01ef01000135","5deb5c8d0ff3bbdf010000be","5deee6dc66e59ade01000150","5def640afe2024de010000d9","5def6c3cfe2024de01000162","5defcc54090769df01000042","5df0d34096fa76de010000dd","5df31dd13b310cde0100008d","5df6d15dda9d91de0100000b","5df71f4bd581f5df010000fb","5df9ce5938a7afde0100002d","5dfae32bfa2bbbde01000015","5dfc1ef5ff6df7de01000042","5dfca63c69d4b0de0100001d","5dfe85ee26fad1de0100002a","5dfef845e25b1bde0100001c","5e010cbe28320fde01000043","5e01810daa04bede01000040","5e030bfff2d70dde0100005f","5e038153d6cccbdf0100006e","5e038795d6cccbdf0100009f","5e062b818e1565f20100011d","5e07bd6aea5f0bdf01000017","5e07d4a6ea5f0bdf01000126","5e08cf5f2389e9de0100009c","5e097971ade67ddf010000b5","5e0a384001dcffde010000b5","5e0a5d4ece3ebce401000100","5e0aff073830a1df010000aa","5e0ca0cc6a5164de01000067","5e0ca3526a5164de01000089","5e0ed07cb68da9de01000063","5e0fd2a12cfae9df0100009d","5e11ce89e49b0bdf010000d0","5e1509f188b10dde0100003f","5e154ac5edfb1ede010000fa","5e16dfcbce54f2df0100013a","5e1792efd954d8df0100003d","5e17b8803fc0b5de010000be","5e1911f6a7672ede01000112","5e1a43353a430ede010000e1","5e1ce4caabed9bde0100016a","5e1e43e22e41a7de01000020","5e1ed0c2875c69df01000022","5e1f3ff59ddd0fde0100001f","5e1f6c37e8f5ddde01000199","5e1fc9952b05b2de010000d9","5e24054c87ef2ade0100005e","5e262fab24c8a6de0100004b","5e269d90f3bb7ddf01000016","5e2792444c3b0dde010000eb","5e27f78e4d75d8de0100006c","5e28cc6d6acacbdf010001ec","5e291aa6a8eed2df010000c8","5e29dc07888177df0100013a","5e2e13e3cc9e90e401000006","5e2e2badcc9e90e401000103","5e2e85d78d5204de01000011","5e3054cf57a222df010000f8","5e3148528cf138de0100005a","5e32796a3957e3df01000014","5e327b833957e3df01000034","5e33a25e17f2c9de010000ca","5e343b010c807ede0100000d","5e346f61a8890cde010001ec","5e347ce1fae8b9de010000cc","5e348f9a53b794de01000020","5e34993653b794de010000f6","5e349e8853b794de0100013c","5e358dd6e5db93df0100000d","5e37053d6568afde010000a4","5e37cdde56571fde0100009e","5e38914d030bcadf01000110","5e38ac3e645ed2de010000ec","5e39759a346d7cde01000170","5e3cf670ad8243de0100008f","5e3db44807ca74de01000059","5e3e1f02546544df0100000b","5e3e7a20666d79df0100003e","5e3f1a4be7f957de01000128","5e3fb047fa32dbde0100006c","5e41d12be7c67ade01000175","5e4200fcebe241de010000bd","5e43b09aa3f5a4de010000cd","5e44247ffdc393de01000209","5e44695b084293df01000068","5e447880084293df01000175","5e482c7908d3f9de010000a0","5e4bf9d40dff2bde0100005e","5e52263dfe135dde01000035","5e53cbc280e18ade01000072","5e540737e81566de010000da","5e5442187a758fde0100010d","5e5573ade11ab9df01000092","5e5ae3bfd85d5dde010000f8","5e5c0f4e82b694df010000fd","5e5d95285726ecdf01000096","5e6076fa9119f0de010000ab","5e6090c81fc211de010000a6","5e62852b6f352ede01000106","5e631184e358e6de01000054","5e6380410ddad9de01000068","5e67702710be53de01000032","5e67eba70e29d3de010001a8","5e6809cec1fce0de010002b8","5e68a48478b561de01000050","5ppRKQTwvCipHjKTj","65DP8sTmKjAypRDhx","68LsbwCrHuZGRZXoR","6QNvbGANDo3PFdEfn","7M7LzH8GDYSKprMrP","7SaDGWPJ4T4y4zR4j","7dMTurr8GwiGiDspE","86rQrhjuyuzb2uTqq","8AgxSDeForYgaGhFP","8EY6y5Ccv6q9fn9iH","8Fju7hq7m33vvy6HX","8SNaYLGL454ik39Sk","8bG2phDPwMEN4Fn2e","8dcm79EqzNLqJbRtc","8qTdmuwNpK69mynWe","AaYrtiimaN6mxHzHg","AafJqQ7MyewrqZsbK","At4dHTWofyPFXbPTE","BG9cYYQtX6QqcAAh9","C7DSMcPmfoeLvgLYT","CBYjYZDrNymq7F3D5","CGepcnvHqasMgzwon","CXAXbPDRyDLHH62Ya","CoKNeetHPJTdFRWNR","Dnjo2TpGFnZw66fmm","Drg2K3pHDbaTEHmzr","Dwnq957u49BCXgJLW","EFgA8RpFc2tYPqt6s","EuJAhupytAhjZoo2c","FK6E4sJ98GB4xciR8","FuZqQXKjiYseS8xRZ","GGHDnfqr6pjyY4cGa","GJ2ChAxYSfTg2crvg","GMqQPxADhjimt2HSv","GcXEX9kpndCQAoKj4","Gm3xTnQMzCMkh9G6b","GwFzKCXtkvHorz3Xk","HHXctgf2oCbMnmFgf","HXEFrFSbWv7u2wBvQ","HiyhR2BwfkKdviksy","J3dB2nzF3Jqbtr8kv","KD538rCuP6F8MH53E","KEjdF5JaDrXypJDdN","KHd7BDQBcAE4D7RRo","KQzdhfo3nosTwaBNG","KcczPH8exrDBurbM6","KfYZEDCKcKPp5FhQN","KxoDi4QgqK948bBJ4","LLe3HLrMLzuZrHzwL","LNGgTzbLHCHbwo64J","Lb3sWT7dNbaGEtft7","Lct8TtpKtGLeD88E3","M4vrm4Te6FH46JNft","MQmyyQTK9QcWyBDmh","MziCZKKrFJRnPyBC9","NDTMJJavXh5JgaCtJ","NMdiWHtmCP9WySW5E","NnPuR33ifS8zHwYSM","PNN4gP82HobmRnWwz","PawWxNNE5BFXPHxTu","Pxt9byGw8ieyp323f","Q5YSkWriSmk9z6rpY","QWFAhXEZXEfNL9yPz","QcvBwACotWFFkqGYf","QtPtc7zaCeNSnduTs","SPcmshnC56CAcYHMm","SQX8zTovne6peqwgT","StDgmdugEb75Q8fNZ","T7NoxNCEAKzSeNunJ","TXYfgSy8iEezqFeuY","TfpHLmLzBiivfBxhD","TsgeXiXcL7zc2NHZ2","TxJYF8DLm7zjraorB","WSAGQicj5eBNKbk8S","XLvshMx8BHfgJuNAc","Xn2jLRCmu2WFswgbq","XtbnN6TvC5frNouqD","YB2wHKSQPipMs37Qd","YKcGRg7GvDkuj5JQ7","YfmR3pNeD65fBKf4H","Yi9aaSuZdqBwGkKNT","YsQfjgT3gvCnYWhkc","ZFdyZEQ6XyuRT8H8Z","ZX4iZgecCFeJSgmJz","ZaAwFJBftKgopyCAc","a9oG8HfnMzGph8kFD","aNMJNDfAPxiKe8nvx","awwYyBcqYzh2A5WH4","bCcdd57PL3aZJo8Kf","bSz2XzbsnZscxH5oE","bYhgXbkQk9W7Km3z2","bfu2KP7syRutTxZfE","bkxGADunHK7e8yxdR","bnYPreurSbFa5FJcT","c7o88ujHtxwZgSwkQ","cBY8KiZX3mRqzFKS6","cDfzaqm2pYKcLSN4f","cFvqwX4tzDwdjXusJ","cnWoK8Xd9dvzRzrvv","cyiwJutGtqB3RFxo4","djW4kByTkN2CmYvsN","fFgp3BFgsg5gbsusx","fWD3koNuNRHJaJHxW","fbwoK7T4JSHosNKvD","fnMQsaZnLWphXNrBJ","fpC9ummK8ZGYCP8NL","fxpTGoxogZcM7a3Rd","gGkoJfJNyoDsn3vTp","gH6n6PLTRJquTzLHo","gi5MrwhxBEhnmHhjG","gvqioiueqE4Mwzy3f","hLyKyAXwpmydcWMq3","iR3PooLpsbLSvrrnZ","ifSMT8DxpxR9b3Rqt","iipYBxhY2vBawcct3","izLf9cQv3MkfGqRCu","iztC32L7fGcy32Gaf","jEkyr9rkPS6PadBL6","jNh6k7CRkCsmZX4sA","jNsNuJ3gBGfiomQR2","jkioj8pk9xYNG65bn","kNm3eiJHrcPvwcmW8","kP2ZZGYMQbRJfFBjM","kiKYybaLaBPk4PfR9","kkPkDM5kbvo3bBG7w","kzguk8fqGibvpkXnA","n2P3nmF9t3astyxWB","n9NrHqgu6dXrzNv5j","nDvzpAfM3LxFyCZqK","o4Aq9FXbxXun9u9ym","oDTLWMhKQcfXPEoKr","oL6Psk6EbhaheCiKA","ohkSNbBHfu3hReYyo","onLdispmZYv4iuQYe","oxyx9Jygb7ffiLaS5","p7Qg9qDsT2SSrzKRX","pK5vvoNPobHX36T4f","pvozakPuXhMFr6SFm","qaTZF3ZdCyH2kRgKJ","qiPbDov69HbgYeBD7","qwStDEh5uLDed6qsA","qzw9GR4QhJEQbmir6","rZDrE8R8HysN5Lurg","snfdwFE25Qn9Kdu4x","sxgJ8XEmp6DtSuE6g","t3BbNpucovGzbE4Y3","t4ZsozDDMQ9rbYMoF","tRTJsdgQDzPpphE78","tS9tXB7cduR6jDyF7","tez6jec7JT6hYDBzG","tjNwTnpdeHHzJajug","u2JeRXLiyTvjJcssD","u6aL5TnBu8r7wJmpP","uB2KeyMdqvm3w8QSi","uJfCExTZQqhjoQPhL","uSji3wbpmDPeqJsfw","uYJdfZJBqyeRAiTjE","uYqyETFXoefYZPsf2","uZMFnWQKx6qLD7Wew","upCLopBDJMvgCwcK7","utRHnsvjNo28iStqS","uxbMLmG37uYTp4trh","v2q58sg4vXJEohBuB","vAQFuLkzFLNZuXyCm","vJgv2kSzEJm2ksAEt","vW26R23gfzBPMEEDW","va4RY4mJXYN2SH6ww","vbhWWMZwjk4pHcEg9","veyCB4QLxCWrAFsNb","wfy38ApaTNKaZrGaF","xJMwoiTwbp5a8ffGc","xkNSc9eafqiZrSFcH","xkgDk7MM5j2WFyp7q","xnRTp8F4Qd6hZWY7N","yAa4h6hovaoG9REaP","yM3GiTgf8Ltfioqew","yzeNNf956SiCeoQQq","z4X33KQXv2DvwZB2H","z5LWQfx9DNqFn7o8y","zjuMhA4aonAF4787F","ztmPxkg2KFDSR6SbW","zy28MnE7cht8fhpqd","zzbgnWqyrSE2tHc5c"],"dataSources":["oLYpAfsADT9uCu5oW","P4wbzpDjKahb4Yawu","Hvc9pNwWw2boxABn6","2RX7MNkKAsdgHwq9X","kxTAWhAJ5AEaGtDRP","W9wiZPEd3CMo9Zvyj","mQx34sdtPSKFkQLhg","wYLr8SBM4nf5YstZT","wGK6ZauNs4mPREmZA","myJ9KFC5zXN4qYWpW","6h4p3KcA7HgFLQpDk","jBaLf3eYjNGbA43MA","hzmGg9XcrhhzCFFLq"]}