The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training. Erhan, D., Manzagol, P., Bengio, Y., Bengio, S., & Vincent, P. In van Dyk, D. & Wellings, M., editors, Proceedings of The Twelfth International Conference on Artificial Intelligence and Statistics, AISTATS, volume 5, of JMLR Workshop and Conference Procedings, pages 153–160, 2009.
The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training [link]Paper  abstract   bibtex   
Whereas theoretical work suggests that deep architectures might be more efficient at representing highly-varying functions, training deep architectures was unsuccessful until the recent advent of algorithms based on unsupervised pre-training. Even though these new algorithms have enabled training deep models, many questions remain as to the nature of this difficult learning problem. Answering these questions is important if learning in deep architectures is to be further improved. We attempt to shed some light on these questions through extensive simulations. The experiments confirm and clarify the advantage of unsupervised pre-training. They demonstrate the robustness of the training procedure with respect to the random initialization, the positive effect of pre-training in terms of optimization and its role as a regularizer. We empirically show the influence of pre-training with respect to architecture depth, model capacity, and number of training examples.
@inproceedings{erhan:2009:aistat,
  author = {D. Erhan and P.-A. Manzagol and Y. Bengio and S. Bengio and P. Vincent},
  title = {The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training},
  booktitle = {Proceedings of The Twelfth International Conference on Artificial Intelligence and Statistics, {AISTATS}},
  editor = {D. van Dyk and M. Wellings},
  year = 2009,
  series = {JMLR Workshop and Conference Procedings},
  volume = 5,
  pages = {153--160},
  url = {publications/ps/erhan_2009_aistat.ps.gz},
  pdf = {publications/pdf/erhan_2009_aistat.pdf},
  djvu = {publications/djvu/erhan_2009_aistat.djvu},
  web = {http://jmlr.csail.mit.edu/proceedings/papers/v5/erhan09a/erhan09a.pdf},
  original = {2009/deep_study_aistats},
  topics = {deep_learning},
  abstract = {Whereas theoretical work suggests that deep architectures might be more efficient at representing highly-varying functions, training deep architectures was unsuccessful until the recent advent of algorithms based on unsupervised pre-training. Even though these new algorithms have enabled training deep models, many questions remain as to the nature of this difficult learning problem. Answering these questions is important if learning in deep architectures is to be further improved. We attempt to shed some light on these questions through extensive simulations. The experiments confirm and clarify the advantage of unsupervised pre-training. They demonstrate the robustness of the training procedure with respect to the random initialization, the positive effect of pre-training in terms of optimization and its role as a regularizer. We empirically show the influence of pre-training with respect to architecture depth, model capacity, and number of training examples.},
  categorie = {C}
}
Downloads: 0
About
Documentation
Keywords
Search
Users
Terms and Conditions of Use
Privacy Policy
Sitemap
© BibBase.org 2021