Is ChatGPT A Good Translator? A Preliminary Study. Jiao, W., Wang, W., Jen-tse, H., Wang, X., & Tu, Z. arXiv.org, January, 2023. Place: Ithaca Publisher: Cornell University Library, arXiv.org
Is ChatGPT A Good Translator? A Preliminary Study [link]Paper  abstract   bibtex   
This report provides a preliminary evaluation of ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness. We adopt the prompts advised by ChatGPT to trigger its translation ability and find that the candidate prompts generally work well and show minor performance differences. By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages. For distant languages, we explore an interesting strategy named \(\mathbf\pivot~prompting\\) that asks ChatGPT to translate the source sentence into a high-resource pivot language before into the target language, which improves the translation performance significantly. As for the translation robustness, ChatGPT does not perform as well as the commercial systems on biomedical abstracts or Reddit comments but is potentially a good translator for spoken language. Scripts and data: https://github.com/wxjiao/Is-ChatGPT-A-Good-Translator
@article{jiao_is_2023,
	title = {Is {ChatGPT} {A} {Good} {Translator}? {A} {Preliminary} {Study}},
	url = {https://www.proquest.com/working-papers/is-chatgpt-good-translator-preliminary-study/docview/2768910094/se-2},
	abstract = {This report provides a preliminary evaluation of ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness. We adopt the prompts advised by ChatGPT to trigger its translation ability and find that the candidate prompts generally work well and show minor performance differences. By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages. For distant languages, we explore an interesting strategy named {\textbackslash}({\textbackslash}mathbf\{pivot{\textasciitilde}prompting\}{\textbackslash}) that asks ChatGPT to translate the source sentence into a high-resource pivot language before into the target language, which improves the translation performance significantly. As for the translation robustness, ChatGPT does not perform as well as the commercial systems on biomedical abstracts or Reddit comments but is potentially a good translator for spoken language. Scripts and data: https://github.com/wxjiao/Is-ChatGPT-A-Good-Translator},
	language = {English},
	journal = {arXiv.org},
	author = {Jiao, Wenxiang and Wang, Wenxuan and Jen-tse, Huang and Wang, Xing and Tu, Zhaopeng},
	month = jan,
	year = {2023},
	note = {Place: Ithaca
Publisher: Cornell University Library, arXiv.org},
	keywords = {Chatbots, Business And Economics--Banking And Finance, Computation and Language, Machine translation, Languages, Robustness, Performance evaluation, Translators},
}

Downloads: 0