Achieving Adversarial Robustness via Sparsity

Achieving Adversarial Robustness via Sparsity. Wang, S., Liao, N., Xiang, L., Ye, N., & Zhang, Q. Machine Learning, 111(2):685–711, February, 2022. arXiv:2009.05423 [cs, stat]

Paper doi abstract bibtex

Network pruning has been known to produce compact models without much accuracy degradation. However, how the pruning process affects a network's robustness and the working mechanism behind remain unresolved. In this work, we theoretically prove that the sparsity of network weights is closely associated with model robustness. Through experiments on a variety of adversarial pruning methods, we find that weights sparsity will not hurt but improve robustness, where both weights inheritance from the lottery ticket and adversarial training improve model robustness in network pruning. Based on these findings, we propose a novel adversarial training method called inverse weights inheritance, which imposes sparse weights distribution on a large network by inheriting weights from a small network, thereby improving the robustness of the large network.

@article{wang_achieving_2022,
	title = {Achieving {Adversarial} {Robustness} via {Sparsity}},
	volume = {111},
	issn = {0885-6125, 1573-0565},
	url = {http://arxiv.org/abs/2009.05423},
	doi = {10.1007/s10994-021-06049-9},
	abstract = {Network pruning has been known to produce compact models without much accuracy degradation. However, how the pruning process affects a network's robustness and the working mechanism behind remain unresolved. In this work, we theoretically prove that the sparsity of network weights is closely associated with model robustness. Through experiments on a variety of adversarial pruning methods, we find that weights sparsity will not hurt but improve robustness, where both weights inheritance from the lottery ticket and adversarial training improve model robustness in network pruning. Based on these findings, we propose a novel adversarial training method called inverse weights inheritance, which imposes sparse weights distribution on a large network by inheriting weights from a small network, thereby improving the robustness of the large network.},
	number = {2},
	urldate = {2022-09-28},
	journal = {Machine Learning},
	author = {Wang, Shufan and Liao, Ningyi and Xiang, Liyao and Ye, Nanyang and Zhang, Quanshi},
	month = feb,
	year = {2022},
	note = {arXiv:2009.05423 [cs, stat]},
	keywords = {Computer Science - Machine Learning, Statistics - Machine Learning},
	pages = {685--711},
}

Downloads: 0

{"_id":"Q2Kv5utbTYwBCiYiC","bibbaseid":"wang-liao-xiang-ye-zhang-achievingadversarialrobustnessviasparsity-2022","author_short":["Wang, S.","Liao, N.","Xiang, L.","Ye, N.","Zhang, Q."],"bibdata":{"bibtype":"article","type":"article","title":"Achieving Adversarial Robustness via Sparsity","volume":"111","issn":"0885-6125, 1573-0565","url":"http://arxiv.org/abs/2009.05423","doi":"10.1007/s10994-021-06049-9","abstract":"Network pruning has been known to produce compact models without much accuracy degradation. However, how the pruning process affects a network's robustness and the working mechanism behind remain unresolved. In this work, we theoretically prove that the sparsity of network weights is closely associated with model robustness. Through experiments on a variety of adversarial pruning methods, we find that weights sparsity will not hurt but improve robustness, where both weights inheritance from the lottery ticket and adversarial training improve model robustness in network pruning. Based on these findings, we propose a novel adversarial training method called inverse weights inheritance, which imposes sparse weights distribution on a large network by inheriting weights from a small network, thereby improving the robustness of the large network.","number":"2","urldate":"2022-09-28","journal":"Machine Learning","author":[{"propositions":[],"lastnames":["Wang"],"firstnames":["Shufan"],"suffixes":[]},{"propositions":[],"lastnames":["Liao"],"firstnames":["Ningyi"],"suffixes":[]},{"propositions":[],"lastnames":["Xiang"],"firstnames":["Liyao"],"suffixes":[]},{"propositions":[],"lastnames":["Ye"],"firstnames":["Nanyang"],"suffixes":[]},{"propositions":[],"lastnames":["Zhang"],"firstnames":["Quanshi"],"suffixes":[]}],"month":"February","year":"2022","note":"arXiv:2009.05423 [cs, stat]","keywords":"Computer Science - Machine Learning, Statistics - Machine Learning","pages":"685–711","bibtex":"@article{wang_achieving_2022,\n\ttitle = {Achieving {Adversarial} {Robustness} via {Sparsity}},\n\tvolume = {111},\n\tissn = {0885-6125, 1573-0565},\n\turl = {http://arxiv.org/abs/2009.05423},\n\tdoi = {10.1007/s10994-021-06049-9},\n\tabstract = {Network pruning has been known to produce compact models without much accuracy degradation. However, how the pruning process affects a network's robustness and the working mechanism behind remain unresolved. In this work, we theoretically prove that the sparsity of network weights is closely associated with model robustness. Through experiments on a variety of adversarial pruning methods, we find that weights sparsity will not hurt but improve robustness, where both weights inheritance from the lottery ticket and adversarial training improve model robustness in network pruning. Based on these findings, we propose a novel adversarial training method called inverse weights inheritance, which imposes sparse weights distribution on a large network by inheriting weights from a small network, thereby improving the robustness of the large network.},\n\tnumber = {2},\n\turldate = {2022-09-28},\n\tjournal = {Machine Learning},\n\tauthor = {Wang, Shufan and Liao, Ningyi and Xiang, Liyao and Ye, Nanyang and Zhang, Quanshi},\n\tmonth = feb,\n\tyear = {2022},\n\tnote = {arXiv:2009.05423 [cs, stat]},\n\tkeywords = {Computer Science - Machine Learning, Statistics - Machine Learning},\n\tpages = {685--711},\n}\n\n","author_short":["Wang, S.","Liao, N.","Xiang, L.","Ye, N.","Zhang, Q."],"key":"wang_achieving_2022","id":"wang_achieving_2022","bibbaseid":"wang-liao-xiang-ye-zhang-achievingadversarialrobustnessviasparsity-2022","role":"author","urls":{"Paper":"http://arxiv.org/abs/2009.05423"},"keyword":["Computer Science - Machine Learning","Statistics - Machine Learning"],"metadata":{"authorlinks":{}},"downloads":0,"html":""},"bibtype":"article","biburl":"https://bibbase.org/zotero/bxt101","dataSources":["Wsv2bQ4jPuc7qme8R"],"keywords":["computer science - machine learning","statistics - machine learning"],"search_terms":["achieving","adversarial","robustness","via","sparsity","wang","liao","xiang","ye","zhang"],"title":"Achieving Adversarial Robustness via Sparsity","year":2022}