Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond. Chang, H., Yao, Z., Gon, A., Yu, H., & McCallum, A. July, 2023. ACL 2023, equal contribution from the first two authors.![Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper abstract bibtex Is the output softmax layer, which is adopted by most language models (LMs), always the best way to compute the next word probability? Given so many attention layers in a modern transformer-based LM, are the pointer networks redundant nowadays? In this study, we discover that the answers to both questions are no. This is because the softmax bottleneck sometimes prevents the LMs from predicting the desired distribution and the pointer networks can be used to break the bottleneck efficiently. Based on the finding, we propose several softmax alternatives by simplifying the pointer networks and accelerating the word-by-word rerankers. In GPT-2, our proposals are significantly better and more efficient than mixture of softmax, a state-of-the-art softmax alternative. In summarization experiments, without significantly decreasing its training/testing speed, our best method based on T5-Small improves factCC score by 2 points in CNN/DM and XSUM dataset, and improves MAUVE scores by 30% in BookSum paragraph-level dataset.
@misc{chang_revisiting_2023,
address = {Canada},
title = {Revisiting the {Architectures} like {Pointer} {Networks} to {Efficiently} {Improve} the {Next} {Word} {Distribution}, {Summarization} {Factuality}, and {Beyond}},
url = {http://arxiv.org/abs/2305.12289},
abstract = {Is the output softmax layer, which is adopted by most language models (LMs), always the best way to compute the next word probability? Given so many attention layers in a modern transformer-based LM, are the pointer networks redundant nowadays? In this study, we discover that the answers to both questions are no. This is because the softmax bottleneck sometimes prevents the LMs from predicting the desired distribution and the pointer networks can be used to break the bottleneck efficiently. Based on the finding, we propose several softmax alternatives by simplifying the pointer networks and accelerating the word-by-word rerankers. In GPT-2, our proposals are significantly better and more efficient than mixture of softmax, a state-of-the-art softmax alternative. In summarization experiments, without significantly decreasing its training/testing speed, our best method based on T5-Small improves factCC score by 2 points in CNN/DM and XSUM dataset, and improves MAUVE scores by 30\% in BookSum paragraph-level dataset.},
urldate = {2023-05-23},
publisher = {arXiv},
author = {Chang, Haw-Shiuan and Yao, Zonghai and Gon, Alolika and Yu, Hong and McCallum, Andrew},
month = jul,
year = {2023},
note = {ACL 2023, equal contribution from the first two authors.},
keywords = {Computer Science - Computation and Language},
}
Downloads: 0
{"_id":"nvrs4Hs7KudNT8ZCA","bibbaseid":"chang-yao-gon-yu-mccallum-revisitingthearchitectureslikepointernetworkstoefficientlyimprovethenextworddistributionsummarizationfactualityandbeyond-2023","author_short":["Chang, H.","Yao, Z.","Gon, A.","Yu, H.","McCallum, A."],"bibdata":{"bibtype":"misc","type":"misc","address":"Canada","title":"Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond","url":"http://arxiv.org/abs/2305.12289","abstract":"Is the output softmax layer, which is adopted by most language models (LMs), always the best way to compute the next word probability? Given so many attention layers in a modern transformer-based LM, are the pointer networks redundant nowadays? In this study, we discover that the answers to both questions are no. This is because the softmax bottleneck sometimes prevents the LMs from predicting the desired distribution and the pointer networks can be used to break the bottleneck efficiently. Based on the finding, we propose several softmax alternatives by simplifying the pointer networks and accelerating the word-by-word rerankers. In GPT-2, our proposals are significantly better and more efficient than mixture of softmax, a state-of-the-art softmax alternative. In summarization experiments, without significantly decreasing its training/testing speed, our best method based on T5-Small improves factCC score by 2 points in CNN/DM and XSUM dataset, and improves MAUVE scores by 30% in BookSum paragraph-level dataset.","urldate":"2023-05-23","publisher":"arXiv","author":[{"propositions":[],"lastnames":["Chang"],"firstnames":["Haw-Shiuan"],"suffixes":[]},{"propositions":[],"lastnames":["Yao"],"firstnames":["Zonghai"],"suffixes":[]},{"propositions":[],"lastnames":["Gon"],"firstnames":["Alolika"],"suffixes":[]},{"propositions":[],"lastnames":["Yu"],"firstnames":["Hong"],"suffixes":[]},{"propositions":[],"lastnames":["McCallum"],"firstnames":["Andrew"],"suffixes":[]}],"month":"July","year":"2023","note":"ACL 2023, equal contribution from the first two authors.","keywords":"Computer Science - Computation and Language","bibtex":"@misc{chang_revisiting_2023,\n\taddress = {Canada},\n\ttitle = {Revisiting the {Architectures} like {Pointer} {Networks} to {Efficiently} {Improve} the {Next} {Word} {Distribution}, {Summarization} {Factuality}, and {Beyond}},\n\turl = {http://arxiv.org/abs/2305.12289},\n\tabstract = {Is the output softmax layer, which is adopted by most language models (LMs), always the best way to compute the next word probability? Given so many attention layers in a modern transformer-based LM, are the pointer networks redundant nowadays? In this study, we discover that the answers to both questions are no. This is because the softmax bottleneck sometimes prevents the LMs from predicting the desired distribution and the pointer networks can be used to break the bottleneck efficiently. Based on the finding, we propose several softmax alternatives by simplifying the pointer networks and accelerating the word-by-word rerankers. In GPT-2, our proposals are significantly better and more efficient than mixture of softmax, a state-of-the-art softmax alternative. In summarization experiments, without significantly decreasing its training/testing speed, our best method based on T5-Small improves factCC score by 2 points in CNN/DM and XSUM dataset, and improves MAUVE scores by 30\\% in BookSum paragraph-level dataset.},\n\turldate = {2023-05-23},\n\tpublisher = {arXiv},\n\tauthor = {Chang, Haw-Shiuan and Yao, Zonghai and Gon, Alolika and Yu, Hong and McCallum, Andrew},\n\tmonth = jul,\n\tyear = {2023},\n\tnote = {ACL 2023, equal contribution from the first two authors.},\n\tkeywords = {Computer Science - Computation and Language},\n}\n\n","author_short":["Chang, H.","Yao, Z.","Gon, A.","Yu, H.","McCallum, A."],"key":"chang_revisiting_2023","id":"chang_revisiting_2023","bibbaseid":"chang-yao-gon-yu-mccallum-revisitingthearchitectureslikepointernetworkstoefficientlyimprovethenextworddistributionsummarizationfactualityandbeyond-2023","role":"author","urls":{"Paper":"http://arxiv.org/abs/2305.12289"},"keyword":["Computer Science - Computation and Language"],"metadata":{"authorlinks":{}},"html":""},"bibtype":"misc","biburl":"http://fenway.cs.uml.edu/papers/pubs-all.bib","dataSources":["TqaA9miSB65nRfS5H"],"keywords":["computer science - computation and language"],"search_terms":["revisiting","architectures","pointer","networks","efficiently","improve","next","word","distribution","summarization","factuality","beyond","chang","yao","gon","yu","mccallum"],"title":"Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond","year":2023}