Counterfactually Probing Language Identity in Multilingual Models. Srinivasan, A., Govindarajan, V. S., & Mahowald, K. October, 2023. arXiv:2310.18862 [cs]
Counterfactually Probing Language Identity in Multilingual Models [link]Paper  doi  abstract   bibtex   
Techniques in causal analysis of language models illuminate how linguistic information is organized in LLMs. We use one such technique, AlterRep, a method of counterfactual probing, to explore the internal structure of multilingual models (mBERT and XLM-R). We train a linear classifier on a binary language identity task, to classify tokens between Language X and Language Y. Applying a counterfactual probing procedure, we use the classifier weights to project the embeddings into the null space and push the resulting embeddings either in the direction of Language X or Language Y. Then we evaluate on a masked language modeling task. We find that, given a template in Language X, pushing towards Language Y systematically increases the probability of Language Y words, above and beyond a third-party control language. But it does not specifically push the model towards translation-equivalent words in Language Y. Pushing towards Language X (the same direction as the template) has a minimal effect, but somewhat degrades these models. Overall, we take these results as further evidence of the rich structure of massive multilingual language models, which include both a language-specific and language-general component. And we show that counterfactual probing can be fruitfully applied to multilingual models.
@misc{srinivasan_counterfactually_2023,
	title = {Counterfactually {Probing} {Language} {Identity} in {Multilingual} {Models}},
	url = {http://arxiv.org/abs/2310.18862},
	doi = {10.48550/arXiv.2310.18862},
	abstract = {Techniques in causal analysis of language models illuminate how linguistic information is organized in LLMs. We use one such technique, AlterRep, a method of counterfactual probing, to explore the internal structure of multilingual models (mBERT and XLM-R). We train a linear classifier on a binary language identity task, to classify tokens between Language X and Language Y. Applying a counterfactual probing procedure, we use the classifier weights to project the embeddings into the null space and push the resulting embeddings either in the direction of Language X or Language Y. Then we evaluate on a masked language modeling task. We find that, given a template in Language X, pushing towards Language Y systematically increases the probability of Language Y words, above and beyond a third-party control language. But it does not specifically push the model towards translation-equivalent words in Language Y. Pushing towards Language X (the same direction as the template) has a minimal effect, but somewhat degrades these models. Overall, we take these results as further evidence of the rich structure of massive multilingual language models, which include both a language-specific and language-general component. And we show that counterfactual probing can be fruitfully applied to multilingual models.},
	urldate = {2025-01-17},
	publisher = {arXiv},
	author = {Srinivasan, Anirudh and Govindarajan, Venkata S. and Mahowald, Kyle},
	month = oct,
	year = {2023},
	note = {arXiv:2310.18862 [cs]},
	keywords = {Computer Science - Computation and Language},
}

Downloads: 0