Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Mendes, C. C. T., Chen, W., Chaudhary, V., Chopra, P., Del Giorno, A., de Rosa, G., Dixon, M., Eldan, R., Iter, D., Garg, A., Goswami, A., Gunasekar, S., Haider, E., Hao, J., Hewett, R. J., Huynh, J., Javaheripi, M., Jin, X., Kauffmann, P., Karampatziakis, N., Kim, D., Khademi, M., Kurilenko, L., Lee, J. R., Lee, Y. T., Li, Y., Liang, C., Liu, W., Lin, E., Lin, Z., Madan, P., Mitra, A., Modi, H., Nguyen, A., Norick, B., Patra, B., Perez-Becker, D., Portet, T., Pryzant, R., Qin, H., Radmilac, M., Rosset, C., Roy, S., Ruwase, O., Saarikivi, O., Saied, A., Salim, A., Santacroce, M., Shah, S., Shang, N., Sharma, H., Song, X., Tanaka, M., Wang, X., Ward, R., Wang, G., Witte, P., Wyatt, M., Xu, C., Xu, J., Yadav, S., Yang, F., Yang, Z., Yu, D., Zhang, C., Zhang, C., Zhang, J., Zhang, L. L., Zhang, Y., Zhang, Y., Zhang, Y., & Zhou, X. April, 2024. arXiv:2404.14219 [cs]

Paper doi abstract bibtex

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. The innovation lies entirely in our dataset for training, a scaled-up version of the one used for phi-2, composed of heavily filtered web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide some initial parameter-scaling results with a 7B and 14B models trained for 4.8T tokens, called phi-3-small and phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75% and 78% on MMLU, and 8.7 and 8.9 on MT-bench).

@misc{abdin_phi-3_2024,
	title = {Phi-3 {Technical} {Report}: {A} {Highly} {Capable} {Language} {Model} {Locally} on {Your} {Phone}},
	shorttitle = {Phi-3 {Technical} {Report}},
	url = {http://arxiv.org/abs/2404.14219},
	doi = {10.48550/arXiv.2404.14219},
	abstract = {We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69\% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. The innovation lies entirely in our dataset for training, a scaled-up version of the one used for phi-2, composed of heavily filtered web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide some initial parameter-scaling results with a 7B and 14B models trained for 4.8T tokens, called phi-3-small and phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75\% and 78\% on MMLU, and 8.7 and 8.9 on MT-bench).},
	urldate = {2024-04-24},
	publisher = {arXiv},
	author = {Abdin, Marah and Jacobs, Sam Ade and Awan, Ammar Ahmad and Aneja, Jyoti and Awadallah, Ahmed and Awadalla, Hany and Bach, Nguyen and Bahree, Amit and Bakhtiari, Arash and Behl, Harkirat and Benhaim, Alon and Bilenko, Misha and Bjorck, Johan and Bubeck, Sébastien and Cai, Martin and Mendes, Caio César Teodoro and Chen, Weizhu and Chaudhary, Vishrav and Chopra, Parul and Del Giorno, Allie and de Rosa, Gustavo and Dixon, Matthew and Eldan, Ronen and Iter, Dan and Garg, Amit and Goswami, Abhishek and Gunasekar, Suriya and Haider, Emman and Hao, Junheng and Hewett, Russell J. and Huynh, Jamie and Javaheripi, Mojan and Jin, Xin and Kauffmann, Piero and Karampatziakis, Nikos and Kim, Dongwoo and Khademi, Mahoud and Kurilenko, Lev and Lee, James R. and Lee, Yin Tat and Li, Yuanzhi and Liang, Chen and Liu, Weishung and Lin, Eric and Lin, Zeqi and Madan, Piyush and Mitra, Arindam and Modi, Hardik and Nguyen, Anh and Norick, Brandon and Patra, Barun and Perez-Becker, Daniel and Portet, Thomas and Pryzant, Reid and Qin, Heyang and Radmilac, Marko and Rosset, Corby and Roy, Sambudha and Ruwase, Olatunji and Saarikivi, Olli and Saied, Amin and Salim, Adil and Santacroce, Michael and Shah, Shital and Shang, Ning and Sharma, Hiteshi and Song, Xia and Tanaka, Masahiro and Wang, Xin and Ward, Rachel and Wang, Guanhua and Witte, Philipp and Wyatt, Michael and Xu, Can and Xu, Jiahang and Yadav, Sonali and Yang, Fan and Yang, Ziyi and Yu, Donghan and Zhang, Chengruidong and Zhang, Cyril and Zhang, Jianwen and Zhang, Li Lyna and Zhang, Yi and Zhang, Yue and Zhang, Yunan and Zhou, Xiren},
	month = apr,
	year = {2024},
	note = {arXiv:2404.14219 [cs]},
}

Downloads: 0