Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics. Cho, H., Sankar, C., Lin, C., Sadagopan, K., Shayandeh, S., Celikyilmaz, A., May, J., & Beirami, A. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5345–5359, Abu Dhabi, United Arab Emirates, December, 2022. Association for Computational Linguistics. Paper abstract bibtex Recent works that revealed the vulnerability of dialogue state tracking (DST) models to distributional shifts have made holistic comparisons on robustness and qualitative analyses increasingly important for understanding their relative performance. We present our findings from standardized and comprehensive DST diagnoses, which have previously been sparse and uncoordinated, using our toolkit, CheckDST, a collection of robustness tests and failure mode analytics. We discover that different classes of DST models have clear strengths and weaknesses, where generation models are more promising for handling language variety while span-based classification models are more robust to unseen entities. Prompted by this discovery, we also compare checkpoints from the same model and find that the standard practice of selecting checkpoints using validation loss/accuracy is prone to overfitting and each model class has distinct patterns of failure. Lastly, we demonstrate how our diagnoses motivate a pre-finetuning procedure with non-dialogue data that offers comprehensive improvements to generation models by alleviating the impact of distributional shifts through transfer learning.
@inproceedings{cho-etal-2022-know,
title = "Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics",
author = "Cho, Hyundong and
Sankar, Chinnadhurai and
Lin, Christopher and
Sadagopan, Kaushik and
Shayandeh, Shahin and
Celikyilmaz, Asli and
May, Jonathan and
Beirami, Ahmad",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.findings-emnlp.391",
pages = "5345--5359",
abstract = "Recent works that revealed the vulnerability of dialogue state tracking (DST) models to distributional shifts have made holistic comparisons on robustness and qualitative analyses increasingly important for understanding their relative performance. We present our findings from standardized and comprehensive DST diagnoses, which have previously been sparse and uncoordinated, using our toolkit, CheckDST, a collection of robustness tests and failure mode analytics. We discover that different classes of DST models have clear strengths and weaknesses, where generation models are more promising for handling language variety while span-based classification models are more robust to unseen entities. Prompted by this discovery, we also compare checkpoints from the same model and find that the standard practice of selecting checkpoints using validation loss/accuracy is prone to overfitting and each model class has distinct patterns of failure. Lastly, we demonstrate how our diagnoses motivate a pre-finetuning procedure with non-dialogue data that offers comprehensive improvements to generation models by alleviating the impact of distributional shifts through transfer learning.",
}
Downloads: 0
{"_id":"sgyc7o2iXEmWCTy7r","bibbaseid":"cho-sankar-lin-sadagopan-shayandeh-celikyilmaz-may-beirami-knowthystrengthscomprehensivedialoguestatetrackingdiagnostics-2022","author_short":["Cho, H.","Sankar, C.","Lin, C.","Sadagopan, K.","Shayandeh, S.","Celikyilmaz, A.","May, J.","Beirami, A."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","title":"Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics","author":[{"propositions":[],"lastnames":["Cho"],"firstnames":["Hyundong"],"suffixes":[]},{"propositions":[],"lastnames":["Sankar"],"firstnames":["Chinnadhurai"],"suffixes":[]},{"propositions":[],"lastnames":["Lin"],"firstnames":["Christopher"],"suffixes":[]},{"propositions":[],"lastnames":["Sadagopan"],"firstnames":["Kaushik"],"suffixes":[]},{"propositions":[],"lastnames":["Shayandeh"],"firstnames":["Shahin"],"suffixes":[]},{"propositions":[],"lastnames":["Celikyilmaz"],"firstnames":["Asli"],"suffixes":[]},{"propositions":[],"lastnames":["May"],"firstnames":["Jonathan"],"suffixes":[]},{"propositions":[],"lastnames":["Beirami"],"firstnames":["Ahmad"],"suffixes":[]}],"booktitle":"Findings of the Association for Computational Linguistics: EMNLP 2022","month":"December","year":"2022","address":"Abu Dhabi, United Arab Emirates","publisher":"Association for Computational Linguistics","url":"https://aclanthology.org/2022.findings-emnlp.391","pages":"5345–5359","abstract":"Recent works that revealed the vulnerability of dialogue state tracking (DST) models to distributional shifts have made holistic comparisons on robustness and qualitative analyses increasingly important for understanding their relative performance. We present our findings from standardized and comprehensive DST diagnoses, which have previously been sparse and uncoordinated, using our toolkit, CheckDST, a collection of robustness tests and failure mode analytics. We discover that different classes of DST models have clear strengths and weaknesses, where generation models are more promising for handling language variety while span-based classification models are more robust to unseen entities. Prompted by this discovery, we also compare checkpoints from the same model and find that the standard practice of selecting checkpoints using validation loss/accuracy is prone to overfitting and each model class has distinct patterns of failure. Lastly, we demonstrate how our diagnoses motivate a pre-finetuning procedure with non-dialogue data that offers comprehensive improvements to generation models by alleviating the impact of distributional shifts through transfer learning.","bibtex":"@inproceedings{cho-etal-2022-know,\n title = \"Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics\",\n author = \"Cho, Hyundong and\n Sankar, Chinnadhurai and\n Lin, Christopher and\n Sadagopan, Kaushik and\n Shayandeh, Shahin and\n Celikyilmaz, Asli and\n May, Jonathan and\n Beirami, Ahmad\",\n booktitle = \"Findings of the Association for Computational Linguistics: EMNLP 2022\",\n month = dec,\n year = \"2022\",\n address = \"Abu Dhabi, United Arab Emirates\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/2022.findings-emnlp.391\",\n pages = \"5345--5359\",\n abstract = \"Recent works that revealed the vulnerability of dialogue state tracking (DST) models to distributional shifts have made holistic comparisons on robustness and qualitative analyses increasingly important for understanding their relative performance. We present our findings from standardized and comprehensive DST diagnoses, which have previously been sparse and uncoordinated, using our toolkit, CheckDST, a collection of robustness tests and failure mode analytics. We discover that different classes of DST models have clear strengths and weaknesses, where generation models are more promising for handling language variety while span-based classification models are more robust to unseen entities. Prompted by this discovery, we also compare checkpoints from the same model and find that the standard practice of selecting checkpoints using validation loss/accuracy is prone to overfitting and each model class has distinct patterns of failure. Lastly, we demonstrate how our diagnoses motivate a pre-finetuning procedure with non-dialogue data that offers comprehensive improvements to generation models by alleviating the impact of distributional shifts through transfer learning.\",\n}\n\n\n","author_short":["Cho, H.","Sankar, C.","Lin, C.","Sadagopan, K.","Shayandeh, S.","Celikyilmaz, A.","May, J.","Beirami, A."],"key":"cho-etal-2022-know","id":"cho-etal-2022-know","bibbaseid":"cho-sankar-lin-sadagopan-shayandeh-celikyilmaz-may-beirami-knowthystrengthscomprehensivedialoguestatetrackingdiagnostics-2022","role":"author","urls":{"Paper":"https://aclanthology.org/2022.findings-emnlp.391"},"metadata":{"authorlinks":{}}},"bibtype":"inproceedings","biburl":"https://jonmay.github.io/webpage/cutelabname/cutelabname.bib","dataSources":["hbZSwot2msWk92m5B","j3Qzx9HAAC6WtJDHS","5eM3sAccSEpjSDHHQ"],"keywords":[],"search_terms":["know","thy","strengths","comprehensive","dialogue","state","tracking","diagnostics","cho","sankar","lin","sadagopan","shayandeh","celikyilmaz","may","beirami"],"title":"Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics","year":2022}