Solving Quantitative Reasoning Problems with Language Models

Solving Quantitative Reasoning Problems with Language Models. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., & Misra, V. June, 2022. arXiv:2206.14858 [cs]

Paper abstract bibtex

Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.

@misc{lewkowycz_solving_2022,
	title = {Solving {Quantitative} {Reasoning} {Problems} with {Language} {Models}},
	url = {http://arxiv.org/abs/2206.14858},
	abstract = {Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.},
	urldate = {2022-07-04},
	publisher = {arXiv},
	author = {Lewkowycz, Aitor and Andreassen, Anders and Dohan, David and Dyer, Ethan and Michalewski, Henryk and Ramasesh, Vinay and Slone, Ambrose and Anil, Cem and Schlag, Imanol and Gutman-Solo, Theo and Wu, Yuhuai and Neyshabur, Behnam and Gur-Ari, Guy and Misra, Vedant},
	month = jun,
	year = {2022},
	note = {arXiv:2206.14858 [cs]},
	keywords = {artificial intelligence, machine learning, uses sympy},
}

Downloads: 0

{"_id":"7aHnh65WsZxb8mXWo","bibbaseid":"lewkowycz-andreassen-dohan-dyer-michalewski-ramasesh-slone-anil-etal-solvingquantitativereasoningproblemswithlanguagemodels-2022","author_short":["Lewkowycz, A.","Andreassen, A.","Dohan, D.","Dyer, E.","Michalewski, H.","Ramasesh, V.","Slone, A.","Anil, C.","Schlag, I.","Gutman-Solo, T.","Wu, Y.","Neyshabur, B.","Gur-Ari, G.","Misra, V."],"bibdata":{"bibtype":"misc","type":"misc","title":"Solving Quantitative Reasoning Problems with Language Models","url":"http://arxiv.org/abs/2206.14858","abstract":"Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.","urldate":"2022-07-04","publisher":"arXiv","author":[{"propositions":[],"lastnames":["Lewkowycz"],"firstnames":["Aitor"],"suffixes":[]},{"propositions":[],"lastnames":["Andreassen"],"firstnames":["Anders"],"suffixes":[]},{"propositions":[],"lastnames":["Dohan"],"firstnames":["David"],"suffixes":[]},{"propositions":[],"lastnames":["Dyer"],"firstnames":["Ethan"],"suffixes":[]},{"propositions":[],"lastnames":["Michalewski"],"firstnames":["Henryk"],"suffixes":[]},{"propositions":[],"lastnames":["Ramasesh"],"firstnames":["Vinay"],"suffixes":[]},{"propositions":[],"lastnames":["Slone"],"firstnames":["Ambrose"],"suffixes":[]},{"propositions":[],"lastnames":["Anil"],"firstnames":["Cem"],"suffixes":[]},{"propositions":[],"lastnames":["Schlag"],"firstnames":["Imanol"],"suffixes":[]},{"propositions":[],"lastnames":["Gutman-Solo"],"firstnames":["Theo"],"suffixes":[]},{"propositions":[],"lastnames":["Wu"],"firstnames":["Yuhuai"],"suffixes":[]},{"propositions":[],"lastnames":["Neyshabur"],"firstnames":["Behnam"],"suffixes":[]},{"propositions":[],"lastnames":["Gur-Ari"],"firstnames":["Guy"],"suffixes":[]},{"propositions":[],"lastnames":["Misra"],"firstnames":["Vedant"],"suffixes":[]}],"month":"June","year":"2022","note":"arXiv:2206.14858 [cs]","keywords":"artificial intelligence, machine learning, uses sympy","bibtex":"@misc{lewkowycz_solving_2022,\n\ttitle = {Solving {Quantitative} {Reasoning} {Problems} with {Language} {Models}},\n\turl = {http://arxiv.org/abs/2206.14858},\n\tabstract = {Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.},\n\turldate = {2022-07-04},\n\tpublisher = {arXiv},\n\tauthor = {Lewkowycz, Aitor and Andreassen, Anders and Dohan, David and Dyer, Ethan and Michalewski, Henryk and Ramasesh, Vinay and Slone, Ambrose and Anil, Cem and Schlag, Imanol and Gutman-Solo, Theo and Wu, Yuhuai and Neyshabur, Behnam and Gur-Ari, Guy and Misra, Vedant},\n\tmonth = jun,\n\tyear = {2022},\n\tnote = {arXiv:2206.14858 [cs]},\n\tkeywords = {artificial intelligence, machine learning, uses sympy},\n}\n\n\n\n\n\n\n\n\n\n\n\n","author_short":["Lewkowycz, A.","Andreassen, A.","Dohan, D.","Dyer, E.","Michalewski, H.","Ramasesh, V.","Slone, A.","Anil, C.","Schlag, I.","Gutman-Solo, T.","Wu, Y.","Neyshabur, B.","Gur-Ari, G.","Misra, V."],"key":"lewkowycz_solving_2022","id":"lewkowycz_solving_2022","bibbaseid":"lewkowycz-andreassen-dohan-dyer-michalewski-ramasesh-slone-anil-etal-solvingquantitativereasoningproblemswithlanguagemodels-2022","role":"author","urls":{"Paper":"http://arxiv.org/abs/2206.14858"},"keyword":["artificial intelligence","machine learning","uses sympy"],"metadata":{"authorlinks":{}}},"bibtype":"misc","biburl":"https://bibbase.org/zotero-group/nicoguaro/525293","dataSources":["YtBDXPDiQEyhyEDZC","fhHfrQgj3AaGp7e9E","qzbMjEJf5d9Lk78vE","45tA9RFoXA9XeH4MM","MeSgs2KDKZo3bEbxH","nSXCrcahhCNfzvXEY","ecatNAsyr4f2iQyGq","tpWeaaCgFjPTYCjg3"],"keywords":["artificial intelligence","machine learning","uses sympy"],"search_terms":["solving","quantitative","reasoning","problems","language","models","lewkowycz","andreassen","dohan","dyer","michalewski","ramasesh","slone","anil","schlag","gutman-solo","wu","neyshabur","gur-ari","misra"],"title":"Solving Quantitative Reasoning Problems with Language Models","year":2022}