Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Linzen, T., Dupoux, E., & Goldberg, Y. Transactions of the Association for Computational Linguistics, 4:521–535, 2016.

Paper abstract bibtex

The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities. Linguistic regularities are often sensitive to syntactic structure; can such dependencies be captured by LSTMs, which do not have explicit structural representations? We begin addressing this question using number agreement in English subject-verb dependencies. We probe the architecture's grammatical competence both using training objectives with an explicit grammatical target (number prediction, grammaticality judgments) and using language models. In the strongly supervised settings, the LSTM achieved very high overall accuracy (less than 1% errors), but errors increased when sequential and structural information conflicted. The frequency of such errors rose sharply in the language-modeling setting. We conclude that LSTMs can capture a non-trivial amount of grammatical structure given targeted supervision, but stronger architectures may be required to further reduce errors; furthermore, the language modeling signal is insufficient for capturing syntax-sensitive dependencies, and should be supplemented with more direct supervision if such dependencies need to be captured.

@article{Linzen2016,
abstract = {The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities. Linguistic regularities are often sensitive to syntactic structure; can such dependencies be captured by LSTMs, which do not have explicit structural representations? We begin addressing this question using number agreement in English subject-verb dependencies. We probe the architecture's grammatical competence both using training objectives with an explicit grammatical target (number prediction, grammaticality judgments) and using language models. In the strongly supervised settings, the LSTM achieved very high overall accuracy (less than 1% errors), but errors increased when sequential and structural information conflicted. The frequency of such errors rose sharply in the language-modeling setting. We conclude that LSTMs can capture a non-trivial amount of grammatical structure given targeted supervision, but stronger architectures may be required to further reduce errors; furthermore, the language modeling signal is insufficient for capturing syntax-sensitive dependencies, and should be supplemented with more direct supervision if such dependencies need to be captured.},
archivePrefix = {arXiv},
arxivId = {1611.01368},
author = {Linzen, Tal and Dupoux, Emmanuel and Goldberg, Yoav},
eprint = {1611.01368},
file = {:Users/shanest/Documents/Library/Linzen, Dupoux, Goldberg/Transactions of the Association for Computational Linguistics/Linzen, Dupoux, Goldberg - 2016 - Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies.pdf:pdf},
journal = {Transactions of the Association for Computational Linguistics},
keywords = {method: psycholinguistic,phenomenon: number agreement},
pages = {521--535},
title = {{Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies}},
url = {http://arxiv.org/abs/1611.01368},
volume = {4},
year = {2016}
}

Downloads: 0

{"_id":"9yymbTRcm4T934Sty","bibbaseid":"linzen-dupoux-goldberg-assessingtheabilityoflstmstolearnsyntaxsensitivedependencies-2016","authorIDs":[],"author_short":["Linzen, T.","Dupoux, E.","Goldberg, Y."],"bibdata":{"bibtype":"article","type":"article","abstract":"The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities. Linguistic regularities are often sensitive to syntactic structure; can such dependencies be captured by LSTMs, which do not have explicit structural representations? We begin addressing this question using number agreement in English subject-verb dependencies. We probe the architecture's grammatical competence both using training objectives with an explicit grammatical target (number prediction, grammaticality judgments) and using language models. In the strongly supervised settings, the LSTM achieved very high overall accuracy (less than 1% errors), but errors increased when sequential and structural information conflicted. The frequency of such errors rose sharply in the language-modeling setting. We conclude that LSTMs can capture a non-trivial amount of grammatical structure given targeted supervision, but stronger architectures may be required to further reduce errors; furthermore, the language modeling signal is insufficient for capturing syntax-sensitive dependencies, and should be supplemented with more direct supervision if such dependencies need to be captured.","archiveprefix":"arXiv","arxivid":"1611.01368","author":[{"propositions":[],"lastnames":["Linzen"],"firstnames":["Tal"],"suffixes":[]},{"propositions":[],"lastnames":["Dupoux"],"firstnames":["Emmanuel"],"suffixes":[]},{"propositions":[],"lastnames":["Goldberg"],"firstnames":["Yoav"],"suffixes":[]}],"eprint":"1611.01368","file":":Users/shanest/Documents/Library/Linzen, Dupoux, Goldberg/Transactions of the Association for Computational Linguistics/Linzen, Dupoux, Goldberg - 2016 - Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies.pdf:pdf","journal":"Transactions of the Association for Computational Linguistics","keywords":"method: psycholinguistic,phenomenon: number agreement","pages":"521–535","title":"Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies","url":"http://arxiv.org/abs/1611.01368","volume":"4","year":"2016","bibtex":"@article{Linzen2016,\nabstract = {The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities. Linguistic regularities are often sensitive to syntactic structure; can such dependencies be captured by LSTMs, which do not have explicit structural representations? We begin addressing this question using number agreement in English subject-verb dependencies. We probe the architecture's grammatical competence both using training objectives with an explicit grammatical target (number prediction, grammaticality judgments) and using language models. In the strongly supervised settings, the LSTM achieved very high overall accuracy (less than 1% errors), but errors increased when sequential and structural information conflicted. The frequency of such errors rose sharply in the language-modeling setting. We conclude that LSTMs can capture a non-trivial amount of grammatical structure given targeted supervision, but stronger architectures may be required to further reduce errors; furthermore, the language modeling signal is insufficient for capturing syntax-sensitive dependencies, and should be supplemented with more direct supervision if such dependencies need to be captured.},\narchivePrefix = {arXiv},\narxivId = {1611.01368},\nauthor = {Linzen, Tal and Dupoux, Emmanuel and Goldberg, Yoav},\neprint = {1611.01368},\nfile = {:Users/shanest/Documents/Library/Linzen, Dupoux, Goldberg/Transactions of the Association for Computational Linguistics/Linzen, Dupoux, Goldberg - 2016 - Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies.pdf:pdf},\njournal = {Transactions of the Association for Computational Linguistics},\nkeywords = {method: psycholinguistic,phenomenon: number agreement},\npages = {521--535},\ntitle = {{Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies}},\nurl = {http://arxiv.org/abs/1611.01368},\nvolume = {4},\nyear = {2016}\n}\n","author_short":["Linzen, T.","Dupoux, E.","Goldberg, Y."],"key":"Linzen2016","id":"Linzen2016","bibbaseid":"linzen-dupoux-goldberg-assessingtheabilityoflstmstolearnsyntaxsensitivedependencies-2016","role":"author","urls":{"Paper":"http://arxiv.org/abs/1611.01368"},"keyword":["method: psycholinguistic","phenomenon: number agreement"],"metadata":{"authorlinks":{}},"downloads":0},"bibtype":"article","biburl":"https://www.shane.st/teaching/575/win20/MachineLearning-interpretability.bib","creationDate":"2020-01-05T04:04:02.901Z","downloads":0,"keywords":["method: psycholinguistic","phenomenon: number agreement"],"search_terms":["assessing","ability","lstms","learn","syntax","sensitive","dependencies","linzen","dupoux","goldberg"],"title":"Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies","year":2016,"dataSources":["okYcdTpf4JJ2zkj7A"]}