Attention is not not Explanation. Wiegreffe, S. & Pinter, Y. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11–20, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics. ![Attention is not not Explanation [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper doi abstract bibtex 1 download Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. Recently, there has been increasing interest in whether or not the intermediate representations offered by these modules may be used to explain the reasoning for a model's prediction, and consequently reach insights regarding the model's decision-making process. A recent paper claims that `Attention is not Explanation' (Jain and Wallace, 2019). We challenge many of the assumptions underlying this work, arguing that such a claim depends on one's definition of explanation, and that testing it needs to take into account all elements of the model, using a rigorous experimental design. We propose four alternative tests to determine when/whether attention can be used as explanation: a simple uniform-weights baseline; a variance calibration based on multiple random seed runs; a diagnostic framework using frozen weights from pretrained models; and an end-to-end adversarial attention training protocol. Each allows for meaningful interpretation of attention mechanisms in RNN models. We show that even when reliable adversarial distributions can be found, they don't perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability.
@inproceedings{Wiegreffe2019,
abstract = {Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. Recently, there has been increasing interest in whether or not the intermediate representations offered by these modules may be used to explain the reasoning for a model's prediction, and consequently reach insights regarding the model's decision-making process. A recent paper claims that `Attention is not Explanation' (Jain and Wallace, 2019). We challenge many of the assumptions underlying this work, arguing that such a claim depends on one's definition of explanation, and that testing it needs to take into account all elements of the model, using a rigorous experimental design. We propose four alternative tests to determine when/whether attention can be used as explanation: a simple uniform-weights baseline; a variance calibration based on multiple random seed runs; a diagnostic framework using frozen weights from pretrained models; and an end-to-end adversarial attention training protocol. Each allows for meaningful interpretation of attention mechanisms in RNN models. We show that even when reliable adversarial distributions can be found, they don't perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability.},
address = {Stroudsburg, PA, USA},
archivePrefix = {arXiv},
arxivId = {1908.04626},
author = {Wiegreffe, Sarah and Pinter, Yuval},
booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
doi = {10.18653/v1/D19-1002},
eprint = {1908.04626},
file = {:Users/shanest/Documents/Library/Wiegreffe, Pinter/Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural./Wiegreffe, Pinter - 2019 - Attention is not not Explanation.pdf:pdf},
keywords = {method: attention},
pages = {11--20},
publisher = {Association for Computational Linguistics},
title = {{Attention is not not Explanation}},
url = {https://www.aclweb.org/anthology/D19-1002},
year = {2019}
}
Downloads: 1
{"_id":"wRrQeNdG8EqKBnnXK","bibbaseid":"wiegreffe-pinter-attentionisnotnotexplanation-2019","authorIDs":[],"author_short":["Wiegreffe, S.","Pinter, Y."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","abstract":"Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. Recently, there has been increasing interest in whether or not the intermediate representations offered by these modules may be used to explain the reasoning for a model's prediction, and consequently reach insights regarding the model's decision-making process. A recent paper claims that `Attention is not Explanation' (Jain and Wallace, 2019). We challenge many of the assumptions underlying this work, arguing that such a claim depends on one's definition of explanation, and that testing it needs to take into account all elements of the model, using a rigorous experimental design. We propose four alternative tests to determine when/whether attention can be used as explanation: a simple uniform-weights baseline; a variance calibration based on multiple random seed runs; a diagnostic framework using frozen weights from pretrained models; and an end-to-end adversarial attention training protocol. Each allows for meaningful interpretation of attention mechanisms in RNN models. We show that even when reliable adversarial distributions can be found, they don't perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability.","address":"Stroudsburg, PA, USA","archiveprefix":"arXiv","arxivid":"1908.04626","author":[{"propositions":[],"lastnames":["Wiegreffe"],"firstnames":["Sarah"],"suffixes":[]},{"propositions":[],"lastnames":["Pinter"],"firstnames":["Yuval"],"suffixes":[]}],"booktitle":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","doi":"10.18653/v1/D19-1002","eprint":"1908.04626","file":":Users/shanest/Documents/Library/Wiegreffe, Pinter/Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural./Wiegreffe, Pinter - 2019 - Attention is not not Explanation.pdf:pdf","keywords":"method: attention","pages":"11–20","publisher":"Association for Computational Linguistics","title":"Attention is not not Explanation","url":"https://www.aclweb.org/anthology/D19-1002","year":"2019","bibtex":"@inproceedings{Wiegreffe2019,\nabstract = {Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. Recently, there has been increasing interest in whether or not the intermediate representations offered by these modules may be used to explain the reasoning for a model's prediction, and consequently reach insights regarding the model's decision-making process. A recent paper claims that `Attention is not Explanation' (Jain and Wallace, 2019). We challenge many of the assumptions underlying this work, arguing that such a claim depends on one's definition of explanation, and that testing it needs to take into account all elements of the model, using a rigorous experimental design. We propose four alternative tests to determine when/whether attention can be used as explanation: a simple uniform-weights baseline; a variance calibration based on multiple random seed runs; a diagnostic framework using frozen weights from pretrained models; and an end-to-end adversarial attention training protocol. Each allows for meaningful interpretation of attention mechanisms in RNN models. We show that even when reliable adversarial distributions can be found, they don't perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability.},\naddress = {Stroudsburg, PA, USA},\narchivePrefix = {arXiv},\narxivId = {1908.04626},\nauthor = {Wiegreffe, Sarah and Pinter, Yuval},\nbooktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},\ndoi = {10.18653/v1/D19-1002},\neprint = {1908.04626},\nfile = {:Users/shanest/Documents/Library/Wiegreffe, Pinter/Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural./Wiegreffe, Pinter - 2019 - Attention is not not Explanation.pdf:pdf},\nkeywords = {method: attention},\npages = {11--20},\npublisher = {Association for Computational Linguistics},\ntitle = {{Attention is not not Explanation}},\nurl = {https://www.aclweb.org/anthology/D19-1002},\nyear = {2019}\n}\n","author_short":["Wiegreffe, S.","Pinter, Y."],"key":"Wiegreffe2019","id":"Wiegreffe2019","bibbaseid":"wiegreffe-pinter-attentionisnotnotexplanation-2019","role":"author","urls":{"Paper":"https://www.aclweb.org/anthology/D19-1002"},"keyword":["method: attention"],"metadata":{"authorlinks":{}},"downloads":1},"bibtype":"inproceedings","biburl":"https://www.shane.st/teaching/575/win20/MachineLearning-interpretability.bib","creationDate":"2020-01-06T00:27:01.034Z","downloads":1,"keywords":["method: attention"],"search_terms":["attention","explanation","wiegreffe","pinter"],"title":"Attention is not not Explanation","year":2019,"dataSources":["okYcdTpf4JJ2zkj7A"]}