On the Cusp of Comprehensibility: Can Language Models Distinguish Between Metaphors and Nonsense?. Griciūt\.e, B., Tanti, M., & Donatelli, L. In Ghosh, D., Beigman Klebanov, B., Muresan, S., Feldman, A., Poria, S., & Chakrabarty, T., editors, Proceedings of the 3rd Workshop on Figurative Language Processing (FLP), pages 173–177, Abu Dhabi, United Arab Emirates (Hybrid), December, 2022. Association for Computational Linguistics. ![On the Cusp of Comprehensibility: Can Language Models Distinguish Between Metaphors and Nonsense? [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper doi abstract bibtex Utterly creative texts can sometimes be difficult to understand, balancing on the edge of comprehensibility. However, good language skills and common sense allow advanced language users both to interpret creative texts and to reject some linguistic input as nonsense. The goal of this paper is to evaluate whether the current language models are also able to make the distinction between a creative language use and nonsense. To test this, we have computed mean rank and pseudo-log-likelihood score (PLL) of metaphorical and nonsensical sentences, and fine-tuned several pretrained models (BERT, RoBERTa) for binary classification between the two categories. There was a significant difference in the mean ranks and PPL scores of the categories, and the classifier reached around 85.5% accuracy. The results raise further questions on what could have let to such satisfactory performance.
@inproceedings{griciute_cusp_2022,
address = {Abu Dhabi, United Arab Emirates (Hybrid)},
title = {On the {Cusp} of {Comprehensibility}: {Can} {Language} {Models} {Distinguish} {Between} {Metaphors} and {Nonsense}?},
shorttitle = {On the {Cusp} of {Comprehensibility}},
url = {https://aclanthology.org/2022.flp-1.25/},
doi = {10.18653/v1/2022.flp-1.25},
abstract = {Utterly creative texts can sometimes be difficult to understand, balancing on the edge of comprehensibility. However, good language skills and common sense allow advanced language users both to interpret creative texts and to reject some linguistic input as nonsense. The goal of this paper is to evaluate whether the current language models are also able to make the distinction between a creative language use and nonsense. To test this, we have computed mean rank and pseudo-log-likelihood score (PLL) of metaphorical and nonsensical sentences, and fine-tuned several pretrained models (BERT, RoBERTa) for binary classification between the two categories. There was a significant difference in the mean ranks and PPL scores of the categories, and the classifier reached around 85.5\% accuracy. The results raise further questions on what could have let to such satisfactory performance.},
urldate = {2025-02-25},
booktitle = {Proceedings of the 3rd {Workshop} on {Figurative} {Language} {Processing} ({FLP})},
publisher = {Association for Computational Linguistics},
author = {Griciūt{\textbackslash}.e, Bernadeta and Tanti, Marc and Donatelli, Lucia},
editor = {Ghosh, Debanjan and Beigman Klebanov, Beata and Muresan, Smaranda and Feldman, Anna and Poria, Soujanya and Chakrabarty, Tuhin},
month = dec,
year = {2022},
pages = {173--177},
}
Downloads: 0
{"_id":"pkYYpjMqj666h6hku","bibbaseid":"gricite-tanti-donatelli-onthecuspofcomprehensibilitycanlanguagemodelsdistinguishbetweenmetaphorsandnonsense-2022","author_short":["Griciūt\\.e, B.","Tanti, M.","Donatelli, L."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","address":"Abu Dhabi, United Arab Emirates (Hybrid)","title":"On the Cusp of Comprehensibility: Can Language Models Distinguish Between Metaphors and Nonsense?","shorttitle":"On the Cusp of Comprehensibility","url":"https://aclanthology.org/2022.flp-1.25/","doi":"10.18653/v1/2022.flp-1.25","abstract":"Utterly creative texts can sometimes be difficult to understand, balancing on the edge of comprehensibility. However, good language skills and common sense allow advanced language users both to interpret creative texts and to reject some linguistic input as nonsense. The goal of this paper is to evaluate whether the current language models are also able to make the distinction between a creative language use and nonsense. To test this, we have computed mean rank and pseudo-log-likelihood score (PLL) of metaphorical and nonsensical sentences, and fine-tuned several pretrained models (BERT, RoBERTa) for binary classification between the two categories. There was a significant difference in the mean ranks and PPL scores of the categories, and the classifier reached around 85.5% accuracy. The results raise further questions on what could have let to such satisfactory performance.","urldate":"2025-02-25","booktitle":"Proceedings of the 3rd Workshop on Figurative Language Processing (FLP)","publisher":"Association for Computational Linguistics","author":[{"propositions":[],"lastnames":["Griciūt\\.e"],"firstnames":["Bernadeta"],"suffixes":[]},{"propositions":[],"lastnames":["Tanti"],"firstnames":["Marc"],"suffixes":[]},{"propositions":[],"lastnames":["Donatelli"],"firstnames":["Lucia"],"suffixes":[]}],"editor":[{"propositions":[],"lastnames":["Ghosh"],"firstnames":["Debanjan"],"suffixes":[]},{"propositions":[],"lastnames":["Beigman","Klebanov"],"firstnames":["Beata"],"suffixes":[]},{"propositions":[],"lastnames":["Muresan"],"firstnames":["Smaranda"],"suffixes":[]},{"propositions":[],"lastnames":["Feldman"],"firstnames":["Anna"],"suffixes":[]},{"propositions":[],"lastnames":["Poria"],"firstnames":["Soujanya"],"suffixes":[]},{"propositions":[],"lastnames":["Chakrabarty"],"firstnames":["Tuhin"],"suffixes":[]}],"month":"December","year":"2022","pages":"173–177","bibtex":"@inproceedings{griciute_cusp_2022,\n\taddress = {Abu Dhabi, United Arab Emirates (Hybrid)},\n\ttitle = {On the {Cusp} of {Comprehensibility}: {Can} {Language} {Models} {Distinguish} {Between} {Metaphors} and {Nonsense}?},\n\tshorttitle = {On the {Cusp} of {Comprehensibility}},\n\turl = {https://aclanthology.org/2022.flp-1.25/},\n\tdoi = {10.18653/v1/2022.flp-1.25},\n\tabstract = {Utterly creative texts can sometimes be difficult to understand, balancing on the edge of comprehensibility. However, good language skills and common sense allow advanced language users both to interpret creative texts and to reject some linguistic input as nonsense. The goal of this paper is to evaluate whether the current language models are also able to make the distinction between a creative language use and nonsense. To test this, we have computed mean rank and pseudo-log-likelihood score (PLL) of metaphorical and nonsensical sentences, and fine-tuned several pretrained models (BERT, RoBERTa) for binary classification between the two categories. There was a significant difference in the mean ranks and PPL scores of the categories, and the classifier reached around 85.5\\% accuracy. The results raise further questions on what could have let to such satisfactory performance.},\n\turldate = {2025-02-25},\n\tbooktitle = {Proceedings of the 3rd {Workshop} on {Figurative} {Language} {Processing} ({FLP})},\n\tpublisher = {Association for Computational Linguistics},\n\tauthor = {Griciūt{\\textbackslash}.e, Bernadeta and Tanti, Marc and Donatelli, Lucia},\n\teditor = {Ghosh, Debanjan and Beigman Klebanov, Beata and Muresan, Smaranda and Feldman, Anna and Poria, Soujanya and Chakrabarty, Tuhin},\n\tmonth = dec,\n\tyear = {2022},\n\tpages = {173--177},\n}\n\n\n\n","author_short":["Griciūt\\.e, B.","Tanti, M.","Donatelli, L."],"editor_short":["Ghosh, D.","Beigman Klebanov, B.","Muresan, S.","Feldman, A.","Poria, S.","Chakrabarty, T."],"key":"griciute_cusp_2022","id":"griciute_cusp_2022","bibbaseid":"gricite-tanti-donatelli-onthecuspofcomprehensibilitycanlanguagemodelsdistinguishbetweenmetaphorsandnonsense-2022","role":"author","urls":{"Paper":"https://aclanthology.org/2022.flp-1.25/"},"metadata":{"authorlinks":{}}},"bibtype":"inproceedings","biburl":"https://bibbase.org/zotero/abhishek-p","dataSources":["h7kKWXpJh2iaX92T5"],"keywords":[],"search_terms":["cusp","comprehensibility","language","models","distinguish","between","metaphors","nonsense","griciūt\\.e","tanti","donatelli"],"title":"On the Cusp of Comprehensibility: Can Language Models Distinguish Between Metaphors and Nonsense?","year":2022}