Optimizing Neural Networks with Kronecker-factored Approximate Curvature. Martens, J. & Grosse, R. arXiv.org, cs.LG, 2015. abstract bibtex We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases is completely non-sparse. It is derived by approximating various large blocks of the Fisher (corresponding to entire layers) as factoring as Kronecker products between two much smaller matrices. While only several times more expensive to compute than the plain stochastic gradient, the updates produced by K-FAC make much more progress optimizing the objective, which results in an algorithm that can be much faster than stochastic gradient descent with momentum in practice. And unlike some previously proposed approximate natural-gradient/Newton methods such as Hessian-free methods, K-FAC works very well in highly stochastic optimization regimes.
@Article{Martens2015,
author = {Martens, James and Grosse, Roger},
title = {Optimizing Neural Networks with Kronecker-factored Approximate Curvature},
journal = {arXiv.org},
volume = {cs.LG},
number = {},
pages = {},
year = {2015},
abstract = {We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network\'s Fisher information matrix which is neither diagonal nor low-rank, and in some cases is completely non-sparse. It is derived by approximating various large blocks of the Fisher (corresponding to entire layers) as factoring as Kronecker products between two much smaller matrices. While only several times more expensive to compute than the plain stochastic gradient, the updates produced by K-FAC make much more progress optimizing the objective, which results in an algorithm that can be much faster than stochastic gradient descent with momentum in practice. And unlike some previously proposed approximate natural-gradient/Newton methods such as Hessian-free methods, K-FAC works very well in highly stochastic optimization regimes.},
location = {},
keywords = {}}
Downloads: 0
{"_id":"nZKpP4QDbHzGmBjz7","bibbaseid":"martens-grosse-optimizingneuralnetworkswithkroneckerfactoredapproximatecurvature-2015","authorIDs":[],"author_short":["Martens, J.","Grosse, R."],"bibdata":{"bibtype":"article","type":"article","author":[{"propositions":[],"lastnames":["Martens"],"firstnames":["James"],"suffixes":[]},{"propositions":[],"lastnames":["Grosse"],"firstnames":["Roger"],"suffixes":[]}],"title":"Optimizing Neural Networks with Kronecker-factored Approximate Curvature","journal":"arXiv.org","volume":"cs.LG","number":"","pages":"","year":"2015","abstract":"We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases is completely non-sparse. It is derived by approximating various large blocks of the Fisher (corresponding to entire layers) as factoring as Kronecker products between two much smaller matrices. While only several times more expensive to compute than the plain stochastic gradient, the updates produced by K-FAC make much more progress optimizing the objective, which results in an algorithm that can be much faster than stochastic gradient descent with momentum in practice. And unlike some previously proposed approximate natural-gradient/Newton methods such as Hessian-free methods, K-FAC works very well in highly stochastic optimization regimes.","location":"","keywords":"","bibtex":"@Article{Martens2015,\nauthor = {Martens, James and Grosse, Roger}, \ntitle = {Optimizing Neural Networks with Kronecker-factored Approximate Curvature}, \njournal = {arXiv.org}, \nvolume = {cs.LG}, \nnumber = {}, \npages = {}, \nyear = {2015}, \nabstract = {We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network\\'s Fisher information matrix which is neither diagonal nor low-rank, and in some cases is completely non-sparse. It is derived by approximating various large blocks of the Fisher (corresponding to entire layers) as factoring as Kronecker products between two much smaller matrices. While only several times more expensive to compute than the plain stochastic gradient, the updates produced by K-FAC make much more progress optimizing the objective, which results in an algorithm that can be much faster than stochastic gradient descent with momentum in practice. And unlike some previously proposed approximate natural-gradient/Newton methods such as Hessian-free methods, K-FAC works very well in highly stochastic optimization regimes.}, \nlocation = {}, \nkeywords = {}}\n\n\n","author_short":["Martens, J.","Grosse, R."],"key":"Martens2015","id":"Martens2015","bibbaseid":"martens-grosse-optimizingneuralnetworkswithkroneckerfactoredapproximatecurvature-2015","role":"author","urls":{},"downloads":0},"bibtype":"article","biburl":"https://gist.githubusercontent.com/stuhlmueller/a37ef2ef4f378ebcb73d249fe0f8377a/raw/6f96f6f779501bd9482896af3e4db4de88c35079/references.bib","creationDate":"2020-01-27T02:13:34.220Z","downloads":0,"keywords":[],"search_terms":["optimizing","neural","networks","kronecker","factored","approximate","curvature","martens","grosse"],"title":"Optimizing Neural Networks with Kronecker-factored Approximate Curvature","year":2015,"dataSources":["hEoKh4ygEAWbAZ5iy"]}