Large Language Model-based Role-Playing for Personalized Medical Jargon Extraction

Large Language Model-based Role-Playing for Personalized Medical Jargon Extraction. Lim, J. H., Kwon, S., Yao, Z., Lalor, J. P., & Yu, H. August, 2024. arXiv:2408.05555 [cs]

Paper abstract bibtex

Previous studies reveal that Electronic Health Records (EHR), which have been widely adopted in the U.S. to allow patients to access their personal medical information, do not have high readability to patients due to the prevalence of medical jargon. Tailoring medical notes to individual comprehension by identifying jargon that is difficult for each person will enhance the utility of generative models. We present the first quantitative analysis to measure the impact of role-playing in LLM in medical term extraction. By comparing the results of Mechanical Turk workers over 20 sentences, our study demonstrates that LLM role-playing improves F1 scores in 95% of cases across 14 different socio-demographic backgrounds. Furthermore, applying role-playing with in-context learning outperformed the previous state-of-the-art models. Our research showed that ChatGPT can improve traditional medical term extraction systems by utilizing role-play to deliver personalized patient education, a potential that previous models had not achieved.

@misc{lim_large_2024,
	title = {Large {Language} {Model}-based {Role}-{Playing} for {Personalized} {Medical} {Jargon} {Extraction}},
	url = {http://arxiv.org/abs/2408.05555},
	abstract = {Previous studies reveal that Electronic Health Records (EHR), which have been widely adopted in the U.S. to allow patients to access their personal medical information, do not have high readability to patients due to the prevalence of medical jargon. Tailoring medical notes to individual comprehension by identifying jargon that is difficult for each person will enhance the utility of generative models. We present the first quantitative analysis to measure the impact of role-playing in LLM in medical term extraction. By comparing the results of Mechanical Turk workers over 20 sentences, our study demonstrates that LLM role-playing improves F1 scores in 95\% of cases across 14 different socio-demographic backgrounds. Furthermore, applying role-playing with in-context learning outperformed the previous state-of-the-art models. Our research showed that ChatGPT can improve traditional medical term extraction systems by utilizing role-play to deliver personalized patient education, a potential that previous models had not achieved.},
	urldate = {2024-09-03},
	publisher = {arXiv},
	author = {Lim, Jung Hoon and Kwon, Sunjae and Yao, Zonghai and Lalor, John P. and Yu, Hong},
	month = aug,
	year = {2024},
	note = {arXiv:2408.05555 [cs]},
	keywords = {Computer Science - Computation and Language},
}

Downloads: 0

{"_id":"KvEMXeGzpHcRnkmug","bibbaseid":"lim-kwon-yao-lalor-yu-largelanguagemodelbasedroleplayingforpersonalizedmedicaljargonextraction-2024","author_short":["Lim, J. H.","Kwon, S.","Yao, Z.","Lalor, J. P.","Yu, H."],"bibdata":{"bibtype":"misc","type":"misc","title":"Large Language Model-based Role-Playing for Personalized Medical Jargon Extraction","url":"http://arxiv.org/abs/2408.05555","abstract":"Previous studies reveal that Electronic Health Records (EHR), which have been widely adopted in the U.S. to allow patients to access their personal medical information, do not have high readability to patients due to the prevalence of medical jargon. Tailoring medical notes to individual comprehension by identifying jargon that is difficult for each person will enhance the utility of generative models. We present the first quantitative analysis to measure the impact of role-playing in LLM in medical term extraction. By comparing the results of Mechanical Turk workers over 20 sentences, our study demonstrates that LLM role-playing improves F1 scores in 95% of cases across 14 different socio-demographic backgrounds. Furthermore, applying role-playing with in-context learning outperformed the previous state-of-the-art models. Our research showed that ChatGPT can improve traditional medical term extraction systems by utilizing role-play to deliver personalized patient education, a potential that previous models had not achieved.","urldate":"2024-09-03","publisher":"arXiv","author":[{"propositions":[],"lastnames":["Lim"],"firstnames":["Jung","Hoon"],"suffixes":[]},{"propositions":[],"lastnames":["Kwon"],"firstnames":["Sunjae"],"suffixes":[]},{"propositions":[],"lastnames":["Yao"],"firstnames":["Zonghai"],"suffixes":[]},{"propositions":[],"lastnames":["Lalor"],"firstnames":["John","P."],"suffixes":[]},{"propositions":[],"lastnames":["Yu"],"firstnames":["Hong"],"suffixes":[]}],"month":"August","year":"2024","note":"arXiv:2408.05555 [cs]","keywords":"Computer Science - Computation and Language","bibtex":"@misc{lim_large_2024,\n\ttitle = {Large {Language} {Model}-based {Role}-{Playing} for {Personalized} {Medical} {Jargon} {Extraction}},\n\turl = {http://arxiv.org/abs/2408.05555},\n\tabstract = {Previous studies reveal that Electronic Health Records (EHR), which have been widely adopted in the U.S. to allow patients to access their personal medical information, do not have high readability to patients due to the prevalence of medical jargon. Tailoring medical notes to individual comprehension by identifying jargon that is difficult for each person will enhance the utility of generative models. We present the first quantitative analysis to measure the impact of role-playing in LLM in medical term extraction. By comparing the results of Mechanical Turk workers over 20 sentences, our study demonstrates that LLM role-playing improves F1 scores in 95\\% of cases across 14 different socio-demographic backgrounds. Furthermore, applying role-playing with in-context learning outperformed the previous state-of-the-art models. Our research showed that ChatGPT can improve traditional medical term extraction systems by utilizing role-play to deliver personalized patient education, a potential that previous models had not achieved.},\n\turldate = {2024-09-03},\n\tpublisher = {arXiv},\n\tauthor = {Lim, Jung Hoon and Kwon, Sunjae and Yao, Zonghai and Lalor, John P. and Yu, Hong},\n\tmonth = aug,\n\tyear = {2024},\n\tnote = {arXiv:2408.05555 [cs]},\n\tkeywords = {Computer Science - Computation and Language},\n}\n\n","author_short":["Lim, J. H.","Kwon, S.","Yao, Z.","Lalor, J. P.","Yu, H."],"key":"lim_large_2024","id":"lim_large_2024","bibbaseid":"lim-kwon-yao-lalor-yu-largelanguagemodelbasedroleplayingforpersonalizedmedicaljargonextraction-2024","role":"author","urls":{"Paper":"http://arxiv.org/abs/2408.05555"},"keyword":["Computer Science - Computation and Language"],"metadata":{"authorlinks":{}},"html":""},"bibtype":"misc","biburl":"http://fenway.cs.uml.edu/papers/pubs-all.bib","dataSources":["TqaA9miSB65nRfS5H"],"keywords":["computer science - computation and language"],"search_terms":["large","language","model","based","role","playing","personalized","medical","jargon","extraction","lim","kwon","yao","lalor","yu"],"title":"Large Language Model-based Role-Playing for Personalized Medical Jargon Extraction","year":2024}