XLNet: Generalized Autoregressive Pretraining for Language Understanding

XLNet: Generalized Autoregressive Pretraining for Language Understanding. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V.

Paper abstract bibtex

With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, XLNet outperforms BERT on 20 tasks, often by a large margin, and achieves state-of-the-art results on 18 tasks including question answering, natural language inference, sentiment analysis, and document ranking.

@article{yangXLNetGeneralizedAutoregressive2019,
  archivePrefix = {arXiv},
  eprinttype = {arxiv},
  eprint = {1906.08237},
  primaryClass = {cs},
  title = {{{XLNet}}: {{Generalized Autoregressive Pretraining}} for {{Language Understanding}}},
  url = {http://arxiv.org/abs/1906.08237},
  shorttitle = {{{XLNet}}},
  abstract = {With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, XLNet outperforms BERT on 20 tasks, often by a large margin, and achieves state-of-the-art results on 18 tasks including question answering, natural language inference, sentiment analysis, and document ranking.},
  urldate = {2019-06-21},
  date = {2019-06-19},
  keywords = {Computer Science - Computation and Language,Computer Science - Machine Learning},
  author = {Yang, Zhilin and Dai, Zihang and Yang, Yiming and Carbonell, Jaime and Salakhutdinov, Ruslan and Le, Quoc V.},
  file = {/home/dimitri/Nextcloud/Zotero/storage/FCUWCRCK/Yang et al. - 2019 - XLNet Generalized Autoregressive Pretraining for .pdf;/home/dimitri/Nextcloud/Zotero/storage/FB7CLIH3/1906.html}
}

Downloads: 0

{"_id":"uiJNssGmdzXfL8gjs","bibbaseid":"yang-dai-yang-carbonell-salakhutdinov-le-xlnetgeneralizedautoregressivepretrainingforlanguageunderstanding","authorIDs":[],"author_short":["Yang, Z.","Dai, Z.","Yang, Y.","Carbonell, J.","Salakhutdinov, R.","Le, Q. V."],"bibdata":{"bibtype":"article","type":"article","archiveprefix":"arXiv","eprinttype":"arxiv","eprint":"1906.08237","primaryclass":"cs","title":"XLNet: Generalized Autoregressive Pretraining for Language Understanding","url":"http://arxiv.org/abs/1906.08237","shorttitle":"XLNet","abstract":"With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, XLNet outperforms BERT on 20 tasks, often by a large margin, and achieves state-of-the-art results on 18 tasks including question answering, natural language inference, sentiment analysis, and document ranking.","urldate":"2019-06-21","date":"2019-06-19","keywords":"Computer Science - Computation and Language,Computer Science - Machine Learning","author":[{"propositions":[],"lastnames":["Yang"],"firstnames":["Zhilin"],"suffixes":[]},{"propositions":[],"lastnames":["Dai"],"firstnames":["Zihang"],"suffixes":[]},{"propositions":[],"lastnames":["Yang"],"firstnames":["Yiming"],"suffixes":[]},{"propositions":[],"lastnames":["Carbonell"],"firstnames":["Jaime"],"suffixes":[]},{"propositions":[],"lastnames":["Salakhutdinov"],"firstnames":["Ruslan"],"suffixes":[]},{"propositions":[],"lastnames":["Le"],"firstnames":["Quoc","V."],"suffixes":[]}],"file":"/home/dimitri/Nextcloud/Zotero/storage/FCUWCRCK/Yang et al. - 2019 - XLNet Generalized Autoregressive Pretraining for .pdf;/home/dimitri/Nextcloud/Zotero/storage/FB7CLIH3/1906.html","bibtex":"@article{yangXLNetGeneralizedAutoregressive2019,\n archivePrefix = {arXiv},\n eprinttype = {arxiv},\n eprint = {1906.08237},\n primaryClass = {cs},\n title = {{{XLNet}}: {{Generalized Autoregressive Pretraining}} for {{Language Understanding}}},\n url = {http://arxiv.org/abs/1906.08237},\n shorttitle = {{{XLNet}}},\n abstract = {With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, XLNet outperforms BERT on 20 tasks, often by a large margin, and achieves state-of-the-art results on 18 tasks including question answering, natural language inference, sentiment analysis, and document ranking.},\n urldate = {2019-06-21},\n date = {2019-06-19},\n keywords = {Computer Science - Computation and Language,Computer Science - Machine Learning},\n author = {Yang, Zhilin and Dai, Zihang and Yang, Yiming and Carbonell, Jaime and Salakhutdinov, Ruslan and Le, Quoc V.},\n file = {/home/dimitri/Nextcloud/Zotero/storage/FCUWCRCK/Yang et al. - 2019 - XLNet Generalized Autoregressive Pretraining for .pdf;/home/dimitri/Nextcloud/Zotero/storage/FB7CLIH3/1906.html}\n}\n\n","author_short":["Yang, Z.","Dai, Z.","Yang, Y.","Carbonell, J.","Salakhutdinov, R.","Le, Q. V."],"key":"yangXLNetGeneralizedAutoregressive2019","id":"yangXLNetGeneralizedAutoregressive2019","bibbaseid":"yang-dai-yang-carbonell-salakhutdinov-le-xlnetgeneralizedautoregressivepretrainingforlanguageunderstanding","role":"author","urls":{"Paper":"http://arxiv.org/abs/1906.08237"},"keyword":["Computer Science - Computation and Language","Computer Science - Machine Learning"],"downloads":0},"bibtype":"article","biburl":"https://raw.githubusercontent.com/dlozeve/newblog/master/bib/all.bib","creationDate":"2020-01-08T20:39:39.378Z","downloads":0,"keywords":["computer science - computation and language","computer science - machine learning"],"search_terms":["xlnet","generalized","autoregressive","pretraining","language","understanding","yang","dai","yang","carbonell","salakhutdinov","le"],"title":"XLNet: Generalized Autoregressive Pretraining for Language Understanding","year":null,"dataSources":["3XqdvqRE7zuX4cm8m"]}