Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey. Wimmer, P., Mehnert, J., & Condurache, A. P. May, 2022. arXiv:2205.08099 [cs, stat]
Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey [link]Paper  doi  abstract   bibtex   
State-of-the-art deep learning models have a parameter count that reaches into the billions. Training, storing and transferring such models is energy and time consuming, thus costly. A big part of these costs is caused by training the network. Model compression lowers storage and transfer costs, and can further make training more efficient by decreasing the number of computations in the forward and/or backward pass. Thus, compressing networks also at training time while maintaining a high performance is an important research topic. This work is a survey on methods which reduce the number of trained weights in deep learning models throughout the training. Most of the introduced methods set network parameters to zero which is called pruning. The presented pruning approaches are categorized into pruning at initialization, lottery tickets and dynamic sparse training. Moreover, we discuss methods that freeze parts of a network at its random initialization. By freezing weights, the number of trainable parameters is shrunken which reduces gradient computations and the dimensionality of the model's optimization space. In this survey we first propose dimensionality reduced training as an underlying mathematical model that covers pruning and freezing during training. Afterwards, we present and discuss different dimensionality reduced training methods.
@misc{wimmer_dimensionality_2022,
	title = {Dimensionality {Reduced} {Training} by {Pruning} and {Freezing} {Parts} of a {Deep} {Neural} {Network}, a {Survey}},
	url = {http://arxiv.org/abs/2205.08099},
	doi = {10.48550/arXiv.2205.08099},
	abstract = {State-of-the-art deep learning models have a parameter count that reaches into the billions. Training, storing and transferring such models is energy and time consuming, thus costly. A big part of these costs is caused by training the network. Model compression lowers storage and transfer costs, and can further make training more efficient by decreasing the number of computations in the forward and/or backward pass. Thus, compressing networks also at training time while maintaining a high performance is an important research topic. This work is a survey on methods which reduce the number of trained weights in deep learning models throughout the training. Most of the introduced methods set network parameters to zero which is called pruning. The presented pruning approaches are categorized into pruning at initialization, lottery tickets and dynamic sparse training. Moreover, we discuss methods that freeze parts of a network at its random initialization. By freezing weights, the number of trainable parameters is shrunken which reduces gradient computations and the dimensionality of the model's optimization space. In this survey we first propose dimensionality reduced training as an underlying mathematical model that covers pruning and freezing during training. Afterwards, we present and discuss different dimensionality reduced training methods.},
	urldate = {2022-10-24},
	publisher = {arXiv},
	author = {Wimmer, Paul and Mehnert, Jens and Condurache, Alexandru Paul},
	month = may,
	year = {2022},
	note = {arXiv:2205.08099 [cs, stat]},
	keywords = {Computer Science - Computer Vision and Pattern Recognition, Computer Science - Machine Learning, Statistics - Machine Learning},
}
Downloads: 0
About
Documentation
Keywords
Search
Users
Terms and Conditions of Use
Privacy Policy
Sitemap
© BibBase.org 2021