Are large language models superhuman chemists?. Mirza, A., Alampara, N., Kunchapu, S., Emoekabu, B., Krishnan, A., Wilhelmi, M., Okereke, M., Eberhardt, J., Elahi, A. M., Greiner, M., Holick, C. T., Gupta, T., Asgari, M., Glaubitz, C., Klepsch, L. C., Köster, Y., Meyer, J., Miret, S., Hoffmann, T., Kreth, F. A., Ringleb, M., Roesner, N., Schubert, U. S., Stafast, L. M., Wonanke, D., Pieler, M., Schwaller, P., & Jablonka, K. M. April, 2024. arXiv:2404.01475 [cond-mat, physics:physics]
Are large language models superhuman chemists? [link]Paper  doi  abstract   bibtex   
Large language models (LLMs) have gained widespread interest due to their ability to process human language and perform tasks on which they have not been explicitly trained. This is relevant for the chemical sciences, which face the problem of small and diverse datasets that are frequently in the form of text. LLMs have shown promise in addressing these issues and are increasingly being harnessed to predict chemical properties, optimize reactions, and even design and conduct experiments autonomously. However, we still have only a very limited systematic understanding of the chemical reasoning capabilities of LLMs, which would be required to improve models and mitigate potential harms. Here, we introduce "ChemBench," an automated framework designed to rigorously evaluate the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of human chemists. We curated more than 7,000 question-answer pairs for a wide array of subfields of the chemical sciences, evaluated leading open and closed-source LLMs, and found that the best models outperformed the best human chemists in our study on average. The models, however, struggle with some chemical reasoning tasks that are easy for human experts and provide overconfident, misleading predictions, such as about chemicals' safety profiles. These findings underscore the dual reality that, although LLMs demonstrate remarkable proficiency in chemical tasks, further research is critical to enhancing their safety and utility in chemical sciences. Our findings also indicate a need for adaptations to chemistry curricula and highlight the importance of continuing to develop evaluation frameworks to improve safe and useful LLMs.
@misc{mirza_are_2024,
	title = {Are large language models superhuman chemists?},
	url = {http://arxiv.org/abs/2404.01475},
	doi = {10.48550/arXiv.2404.01475},
	abstract = {Large language models (LLMs) have gained widespread interest due to their ability to process human language and perform tasks on which they have not been explicitly trained. This is relevant for the chemical sciences, which face the problem of small and diverse datasets that are frequently in the form of text. LLMs have shown promise in addressing these issues and are increasingly being harnessed to predict chemical properties, optimize reactions, and even design and conduct experiments autonomously. However, we still have only a very limited systematic understanding of the chemical reasoning capabilities of LLMs, which would be required to improve models and mitigate potential harms. Here, we introduce "ChemBench," an automated framework designed to rigorously evaluate the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of human chemists. We curated more than 7,000 question-answer pairs for a wide array of subfields of the chemical sciences, evaluated leading open and closed-source LLMs, and found that the best models outperformed the best human chemists in our study on average. The models, however, struggle with some chemical reasoning tasks that are easy for human experts and provide overconfident, misleading predictions, such as about chemicals' safety profiles. These findings underscore the dual reality that, although LLMs demonstrate remarkable proficiency in chemical tasks, further research is critical to enhancing their safety and utility in chemical sciences. Our findings also indicate a need for adaptations to chemistry curricula and highlight the importance of continuing to develop evaluation frameworks to improve safe and useful LLMs.},
	urldate = {2024-04-13},
	publisher = {arXiv},
	author = {Mirza, Adrian and Alampara, Nawaf and Kunchapu, Sreekanth and Emoekabu, Benedict and Krishnan, Aswanth and Wilhelmi, Mara and Okereke, Macjonathan and Eberhardt, Juliane and Elahi, Amir Mohammad and Greiner, Maximilian and Holick, Caroline T. and Gupta, Tanya and Asgari, Mehrdad and Glaubitz, Christina and Klepsch, Lea C. and Köster, Yannik and Meyer, Jakob and Miret, Santiago and Hoffmann, Tim and Kreth, Fabian Alexander and Ringleb, Michael and Roesner, Nicole and Schubert, Ulrich S. and Stafast, Leanne M. and Wonanke, Dinga and Pieler, Michael and Schwaller, Philippe and Jablonka, Kevin Maik},
	month = apr,
	year = {2024},
	note = {arXiv:2404.01475 [cond-mat, physics:physics]},
	keywords = {Computer Science - Artificial Intelligence, Computer Science - Machine Learning, Condensed Matter - Materials Science, Physics - Chemical Physics},
}
Downloads: 0
About
Documentation
Keywords
Search
Users
Terms and Conditions of Use
Privacy Policy
Sitemap
© BibBase.org 2021