The unreasonable effectiveness of the forget gate

The unreasonable effectiveness of the forget gate. van der Westhuizen, J. & Lasenby, J. arXiv:1804.04849 [cs, stat], September, 2018. arXiv: 1804.04849

Paper abstract bibtex

Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chronoinitialized biases, not only provides computational savings but outperforms the standard LSTM on multiple benchmark datasets and competes with some of the best contemporary models. Our proposed network, the JANET, achieves accuracies of 99% and 92.5% on the MNIST and pMNIST datasets, outperforming the standard LSTM which yields accuracies of 98.5% and 91%.

@article{van_der_westhuizen_unreasonable_2018,
	title = {The unreasonable effectiveness of the forget gate},
	url = {http://arxiv.org/abs/1804.04849},
	abstract = {Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chronoinitialized biases, not only provides computational savings but outperforms the standard LSTM on multiple benchmark datasets and competes with some of the best contemporary models. Our proposed network, the JANET, achieves accuracies of 99\% and 92.5\% on the MNIST and pMNIST datasets, outperforming the standard LSTM which yields accuracies of 98.5\% and 91\%.},
	language = {en},
	urldate = {2022-01-19},
	journal = {arXiv:1804.04849 [cs, stat]},
	author = {van der Westhuizen, Jos and Lasenby, Joan},
	month = sep,
	year = {2018},
	note = {arXiv: 1804.04849},
	keywords = {/unread, Computer Science - Machine Learning, Computer Science - Neural and Evolutionary Computing, Statistics - Machine Learning, ⛔ No DOI found},
}

Downloads: 0

{"_id":"6MKAHLrQ5qZuG9jPx","bibbaseid":"vanderwesthuizen-lasenby-theunreasonableeffectivenessoftheforgetgate-2018","authorIDs":[],"author_short":["van der Westhuizen, J.","Lasenby, J."],"bibdata":{"bibtype":"article","type":"article","title":"The unreasonable effectiveness of the forget gate","url":"http://arxiv.org/abs/1804.04849","abstract":"Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chronoinitialized biases, not only provides computational savings but outperforms the standard LSTM on multiple benchmark datasets and competes with some of the best contemporary models. Our proposed network, the JANET, achieves accuracies of 99% and 92.5% on the MNIST and pMNIST datasets, outperforming the standard LSTM which yields accuracies of 98.5% and 91%.","language":"en","urldate":"2022-01-19","journal":"arXiv:1804.04849 [cs, stat]","author":[{"propositions":["van","der"],"lastnames":["Westhuizen"],"firstnames":["Jos"],"suffixes":[]},{"propositions":[],"lastnames":["Lasenby"],"firstnames":["Joan"],"suffixes":[]}],"month":"September","year":"2018","note":"arXiv: 1804.04849","keywords":"/unread, Computer Science - Machine Learning, Computer Science - Neural and Evolutionary Computing, Statistics - Machine Learning, ⛔ No DOI found","bibtex":"@article{van_der_westhuizen_unreasonable_2018,\n\ttitle = {The unreasonable effectiveness of the forget gate},\n\turl = {http://arxiv.org/abs/1804.04849},\n\tabstract = {Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chronoinitialized biases, not only provides computational savings but outperforms the standard LSTM on multiple benchmark datasets and competes with some of the best contemporary models. Our proposed network, the JANET, achieves accuracies of 99\\% and 92.5\\% on the MNIST and pMNIST datasets, outperforming the standard LSTM which yields accuracies of 98.5\\% and 91\\%.},\n\tlanguage = {en},\n\turldate = {2022-01-19},\n\tjournal = {arXiv:1804.04849 [cs, stat]},\n\tauthor = {van der Westhuizen, Jos and Lasenby, Joan},\n\tmonth = sep,\n\tyear = {2018},\n\tnote = {arXiv: 1804.04849},\n\tkeywords = {/unread, Computer Science - Machine Learning, Computer Science - Neural and Evolutionary Computing, Statistics - Machine Learning, ⛔ No DOI found},\n}\n\n","author_short":["van der Westhuizen, J.","Lasenby, J."],"key":"van_der_westhuizen_unreasonable_2018","id":"van_der_westhuizen_unreasonable_2018","bibbaseid":"vanderwesthuizen-lasenby-theunreasonableeffectivenessoftheforgetgate-2018","role":"author","urls":{"Paper":"http://arxiv.org/abs/1804.04849"},"keyword":["/unread","Computer Science - Machine Learning","Computer Science - Neural and Evolutionary Computing","Statistics - Machine Learning","⛔ No DOI found"],"metadata":{"authorlinks":{}},"html":""},"bibtype":"article","biburl":"https://bibbase.org/zotero/victorjhu","creationDate":"2020-01-27T02:13:33.902Z","downloads":0,"keywords":["/unread","computer science - machine learning","computer science - neural and evolutionary computing","statistics - machine learning","⛔ no doi found"],"search_terms":["unreasonable","effectiveness","forget","gate","van der westhuizen","lasenby"],"title":"The unreasonable effectiveness of the forget gate","year":2018,"dataSources":["hEoKh4ygEAWbAZ5iy","CmHEoydhafhbkXXt5"]}