Character-Level Language Modeling with Deeper Self-Attention. Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. arXiv:1808.04444 [cs, stat], August, 2018. arXiv: 1808.04444![Character-Level Language Modeling with Deeper Self-Attention [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper abstract bibtex LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.
@article{al-rfou_character-level_2018,
title = {Character-{Level} {Language} {Modeling} with {Deeper} {Self}-{Attention}},
url = {http://arxiv.org/abs/1808.04444},
abstract = {LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.},
urldate = {2019-10-22},
journal = {arXiv:1808.04444 [cs, stat]},
author = {Al-Rfou, Rami and Choe, Dokook and Constant, Noah and Guo, Mandy and Jones, Llion},
month = aug,
year = {2018},
note = {arXiv: 1808.04444},
keywords = {Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Computer Science - Machine Learning, Statistics - Machine Learning},
}
Downloads: 0
{"_id":"WzBrixLtBcXcq4uyc","bibbaseid":"alrfou-choe-constant-guo-jones-characterlevellanguagemodelingwithdeeperselfattention-2018","authorIDs":[],"author_short":["Al-Rfou, R.","Choe, D.","Constant, N.","Guo, M.","Jones, L."],"bibdata":{"bibtype":"article","type":"article","title":"Character-Level Language Modeling with Deeper Self-Attention","url":"http://arxiv.org/abs/1808.04444","abstract":"LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.","urldate":"2019-10-22","journal":"arXiv:1808.04444 [cs, stat]","author":[{"propositions":[],"lastnames":["Al-Rfou"],"firstnames":["Rami"],"suffixes":[]},{"propositions":[],"lastnames":["Choe"],"firstnames":["Dokook"],"suffixes":[]},{"propositions":[],"lastnames":["Constant"],"firstnames":["Noah"],"suffixes":[]},{"propositions":[],"lastnames":["Guo"],"firstnames":["Mandy"],"suffixes":[]},{"propositions":[],"lastnames":["Jones"],"firstnames":["Llion"],"suffixes":[]}],"month":"August","year":"2018","note":"arXiv: 1808.04444","keywords":"Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Computer Science - Machine Learning, Statistics - Machine Learning","bibtex":"@article{al-rfou_character-level_2018,\n\ttitle = {Character-{Level} {Language} {Modeling} with {Deeper} {Self}-{Attention}},\n\turl = {http://arxiv.org/abs/1808.04444},\n\tabstract = {LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.},\n\turldate = {2019-10-22},\n\tjournal = {arXiv:1808.04444 [cs, stat]},\n\tauthor = {Al-Rfou, Rami and Choe, Dokook and Constant, Noah and Guo, Mandy and Jones, Llion},\n\tmonth = aug,\n\tyear = {2018},\n\tnote = {arXiv: 1808.04444},\n\tkeywords = {Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Computer Science - Machine Learning, Statistics - Machine Learning},\n}\n\n","author_short":["Al-Rfou, R.","Choe, D.","Constant, N.","Guo, M.","Jones, L."],"key":"al-rfou_character-level_2018","id":"al-rfou_character-level_2018","bibbaseid":"alrfou-choe-constant-guo-jones-characterlevellanguagemodelingwithdeeperselfattention-2018","role":"author","urls":{"Paper":"http://arxiv.org/abs/1808.04444"},"keyword":["Computer Science - Artificial Intelligence","Computer Science - Computation and Language","Computer Science - Machine Learning","Statistics - Machine Learning"],"downloads":0,"html":""},"bibtype":"article","biburl":"https://bibbase.org/zotero/franciscoperezsorrosal","creationDate":"2021-02-12T21:37:01.809Z","downloads":0,"keywords":["computer science - artificial intelligence","computer science - computation and language","computer science - machine learning","statistics - machine learning"],"search_terms":["character","level","language","modeling","deeper","self","attention","al-rfou","choe","constant","guo","jones"],"title":"Character-Level Language Modeling with Deeper Self-Attention","year":2018,"dataSources":["qLJ7Ld8T2ZKybATHB"]}