Know Where You're Going: Meta-Learning for Parameter-Efficient Fine-Tuning

Know Where You're Going: Meta-Learning for Parameter-Efficient Fine-Tuning. Gheini, M., Ma, X., & May, J. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11602–11612, Toronto, Canada, July, 2023. Association for Computational Linguistics.

Paper doi abstract bibtex

A recent family of techniques, dubbed lightweight fine-tuning methods, facilitates parameter-efficient transfer by updating only a small set of additional parameters while keeping the parameters of the original model frozen. While proven to be an effective approach, there are no existing studies on if and how such knowledge of the downstream fine-tuning approach calls for complementary measures after pre-training and before fine-tuning. In this work, we show that taking the ultimate choice of fine-tuning into consideration boosts the performance of parameter-efficient fine-tuning. By relying on optimization-based meta-learning using MAML with certain modifications for our distinct purpose, we prime the pre-trained model specifically for parameter-efficient fine-tuning, resulting in gains of up to 4.96 points on cross-lingual NER fine-tuning. Our ablation settings and analyses further reveal that the specific approach we take to meta-learning is crucial for the attained gains.

@inproceedings{gheini-etal-2023-know,
    title = "Know Where You{'}re Going: Meta-Learning for Parameter-Efficient Fine-Tuning",
    author = "Gheini, Mozhdeh  and
      Ma, Xuezhe  and
      May, Jonathan",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.737",
    doi = "10.18653/v1/2023.findings-acl.737",
    pages = "11602--11612",
    abstract = "A recent family of techniques, dubbed lightweight fine-tuning methods, facilitates parameter-efficient transfer by updating only a small set of additional parameters while keeping the parameters of the original model frozen. While proven to be an effective approach, there are no existing studies on if and how such knowledge of the downstream fine-tuning approach calls for complementary measures after pre-training and before fine-tuning. In this work, we show that taking the ultimate choice of fine-tuning into consideration boosts the performance of parameter-efficient fine-tuning. By relying on optimization-based meta-learning using MAML with certain modifications for our distinct purpose, we prime the pre-trained model specifically for parameter-efficient fine-tuning, resulting in gains of up to 4.96 points on cross-lingual NER fine-tuning. Our ablation settings and analyses further reveal that the specific approach we take to meta-learning is crucial for the attained gains.",
}

Downloads: 0

{"_id":"wdMcEvXziwKbLmpJX","bibbaseid":"gheini-ma-may-knowwhereyouregoingmetalearningforparameterefficientfinetuning-2023","author_short":["Gheini, M.","Ma, X.","May, J."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","title":"Know Where You're Going: Meta-Learning for Parameter-Efficient Fine-Tuning","author":[{"propositions":[],"lastnames":["Gheini"],"firstnames":["Mozhdeh"],"suffixes":[]},{"propositions":[],"lastnames":["Ma"],"firstnames":["Xuezhe"],"suffixes":[]},{"propositions":[],"lastnames":["May"],"firstnames":["Jonathan"],"suffixes":[]}],"booktitle":"Findings of the Association for Computational Linguistics: ACL 2023","month":"July","year":"2023","address":"Toronto, Canada","publisher":"Association for Computational Linguistics","url":"https://aclanthology.org/2023.findings-acl.737","doi":"10.18653/v1/2023.findings-acl.737","pages":"11602–11612","abstract":"A recent family of techniques, dubbed lightweight fine-tuning methods, facilitates parameter-efficient transfer by updating only a small set of additional parameters while keeping the parameters of the original model frozen. While proven to be an effective approach, there are no existing studies on if and how such knowledge of the downstream fine-tuning approach calls for complementary measures after pre-training and before fine-tuning. In this work, we show that taking the ultimate choice of fine-tuning into consideration boosts the performance of parameter-efficient fine-tuning. By relying on optimization-based meta-learning using MAML with certain modifications for our distinct purpose, we prime the pre-trained model specifically for parameter-efficient fine-tuning, resulting in gains of up to 4.96 points on cross-lingual NER fine-tuning. Our ablation settings and analyses further reveal that the specific approach we take to meta-learning is crucial for the attained gains.","bibtex":"@inproceedings{gheini-etal-2023-know,\n title = \"Know Where You{'}re Going: Meta-Learning for Parameter-Efficient Fine-Tuning\",\n author = \"Gheini, Mozhdeh and\n Ma, Xuezhe and\n May, Jonathan\",\n booktitle = \"Findings of the Association for Computational Linguistics: ACL 2023\",\n month = jul,\n year = \"2023\",\n address = \"Toronto, Canada\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/2023.findings-acl.737\",\n doi = \"10.18653/v1/2023.findings-acl.737\",\n pages = \"11602--11612\",\n abstract = \"A recent family of techniques, dubbed lightweight fine-tuning methods, facilitates parameter-efficient transfer by updating only a small set of additional parameters while keeping the parameters of the original model frozen. While proven to be an effective approach, there are no existing studies on if and how such knowledge of the downstream fine-tuning approach calls for complementary measures after pre-training and before fine-tuning. In this work, we show that taking the ultimate choice of fine-tuning into consideration boosts the performance of parameter-efficient fine-tuning. By relying on optimization-based meta-learning using MAML with certain modifications for our distinct purpose, we prime the pre-trained model specifically for parameter-efficient fine-tuning, resulting in gains of up to 4.96 points on cross-lingual NER fine-tuning. Our ablation settings and analyses further reveal that the specific approach we take to meta-learning is crucial for the attained gains.\",\n}\n\n","author_short":["Gheini, M.","Ma, X.","May, J."],"key":"gheini-etal-2023-know","id":"gheini-etal-2023-know","bibbaseid":"gheini-ma-may-knowwhereyouregoingmetalearningforparameterefficientfinetuning-2023","role":"author","urls":{"Paper":"https://aclanthology.org/2023.findings-acl.737"},"metadata":{"authorlinks":{}}},"bibtype":"inproceedings","biburl":"https://jonmay.github.io/webpage/cutelabname/cutelabname.bib","dataSources":["j3Qzx9HAAC6WtJDHS","5eM3sAccSEpjSDHHQ"],"keywords":[],"search_terms":["know","going","meta","learning","parameter","efficient","fine","tuning","gheini","ma","may"],"title":"Know Where You're Going: Meta-Learning for Parameter-Efficient Fine-Tuning","year":2023}