Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds. Lalor, J. P., Wu, H., & Yu, H. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4240–4250, Hong Kong, China, November, 2019. Association for Computational Linguistics. NIHMSID: NIHMS1059054![Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper doi abstract bibtex Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human response pattern (RP) data, presenting a significant bottleneck for large data sets like those required for training deep neural networks (DNNs). In this work we propose learning IRT models using RPs generated from artificial crowds of DNN models. We demonstrate the effectiveness of learning IRT models using DNN-generated data through quantitative and qualitative analyses for two NLP tasks. Parameters learned from human and machine RPs for natural language inference and sentiment analysis exhibit medium to large positive correlations. We demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods. Finally, we highlight cases where human expectation about item difficulty does not match difficulty as estimated from the machine RPs.
@inproceedings{lalor_learning_2019,
address = {Hong Kong, China},
title = {Learning {Latent} {Parameters} without {Human} {Response} {Patterns}: {Item} {Response} {Theory} with {Artificial} {Crowds}},
shorttitle = {Learning {Latent} {Parameters} without {Human} {Response} {Patterns}},
url = {https://www.aclweb.org/anthology/D19-1434},
doi = {10.18653/v1/D19-1434},
abstract = {Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human response pattern (RP) data, presenting a significant bottleneck for large data sets like those required for training deep neural networks (DNNs). In this work we propose learning IRT models using RPs generated from artificial crowds of DNN models. We demonstrate the effectiveness of learning IRT models using DNN-generated data through quantitative and qualitative analyses for two NLP tasks. Parameters learned from human and machine RPs for natural language inference and sentiment analysis exhibit medium to large positive correlations. We demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods. Finally, we highlight cases where human expectation about item difficulty does not match difficulty as estimated from the machine RPs.},
urldate = {2019-11-11},
booktitle = {Proceedings of the 2019 {Conference} on {Empirical} {Methods} in {Natural} {Language} {Processing} and the 9th {International} {Joint} {Conference} on {Natural} {Language} {Processing} ({EMNLP}-{IJCNLP})},
publisher = {Association for Computational Linguistics},
author = {Lalor, John P. and Wu, Hao and Yu, Hong},
month = nov,
year = {2019},
pmcid = {PMC6892593},
pmid = {31803865},
note = {NIHMSID: NIHMS1059054},
pages = {4240--4250},
}
Downloads: 0
{"_id":"TXDcuyEt4gWoKagBx","bibbaseid":"lalor-wu-yu-learninglatentparameterswithouthumanresponsepatternsitemresponsetheorywithartificialcrowds-2019","author_short":["Lalor, J. P.","Wu, H.","Yu, H."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","address":"Hong Kong, China","title":"Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds","shorttitle":"Learning Latent Parameters without Human Response Patterns","url":"https://www.aclweb.org/anthology/D19-1434","doi":"10.18653/v1/D19-1434","abstract":"Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human response pattern (RP) data, presenting a significant bottleneck for large data sets like those required for training deep neural networks (DNNs). In this work we propose learning IRT models using RPs generated from artificial crowds of DNN models. We demonstrate the effectiveness of learning IRT models using DNN-generated data through quantitative and qualitative analyses for two NLP tasks. Parameters learned from human and machine RPs for natural language inference and sentiment analysis exhibit medium to large positive correlations. We demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods. Finally, we highlight cases where human expectation about item difficulty does not match difficulty as estimated from the machine RPs.","urldate":"2019-11-11","booktitle":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","publisher":"Association for Computational Linguistics","author":[{"propositions":[],"lastnames":["Lalor"],"firstnames":["John","P."],"suffixes":[]},{"propositions":[],"lastnames":["Wu"],"firstnames":["Hao"],"suffixes":[]},{"propositions":[],"lastnames":["Yu"],"firstnames":["Hong"],"suffixes":[]}],"month":"November","year":"2019","pmcid":"PMC6892593","pmid":"31803865","note":"NIHMSID: NIHMS1059054","pages":"4240–4250","bibtex":"@inproceedings{lalor_learning_2019,\n\taddress = {Hong Kong, China},\n\ttitle = {Learning {Latent} {Parameters} without {Human} {Response} {Patterns}: {Item} {Response} {Theory} with {Artificial} {Crowds}},\n\tshorttitle = {Learning {Latent} {Parameters} without {Human} {Response} {Patterns}},\n\turl = {https://www.aclweb.org/anthology/D19-1434},\n\tdoi = {10.18653/v1/D19-1434},\n\tabstract = {Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human response pattern (RP) data, presenting a significant bottleneck for large data sets like those required for training deep neural networks (DNNs). In this work we propose learning IRT models using RPs generated from artificial crowds of DNN models. We demonstrate the effectiveness of learning IRT models using DNN-generated data through quantitative and qualitative analyses for two NLP tasks. Parameters learned from human and machine RPs for natural language inference and sentiment analysis exhibit medium to large positive correlations. We demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods. Finally, we highlight cases where human expectation about item difficulty does not match difficulty as estimated from the machine RPs.},\n\turldate = {2019-11-11},\n\tbooktitle = {Proceedings of the 2019 {Conference} on {Empirical} {Methods} in {Natural} {Language} {Processing} and the 9th {International} {Joint} {Conference} on {Natural} {Language} {Processing} ({EMNLP}-{IJCNLP})},\n\tpublisher = {Association for Computational Linguistics},\n\tauthor = {Lalor, John P. and Wu, Hao and Yu, Hong},\n\tmonth = nov,\n\tyear = {2019},\n\tpmcid = {PMC6892593},\n\tpmid = {31803865},\n\tnote = {NIHMSID: NIHMS1059054},\n\tpages = {4240--4250},\n}\n\n","author_short":["Lalor, J. P.","Wu, H.","Yu, H."],"key":"lalor_learning_2019","id":"lalor_learning_2019","bibbaseid":"lalor-wu-yu-learninglatentparameterswithouthumanresponsepatternsitemresponsetheorywithartificialcrowds-2019","role":"author","urls":{"Paper":"https://www.aclweb.org/anthology/D19-1434"},"metadata":{"authorlinks":{}},"html":""},"bibtype":"inproceedings","biburl":"http://fenway.cs.uml.edu/papers/pubs-all.bib","dataSources":["TqaA9miSB65nRfS5H"],"keywords":[],"search_terms":["learning","latent","parameters","without","human","response","patterns","item","response","theory","artificial","crowds","lalor","wu","yu"],"title":"Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds","year":2019}