Learning Deep Representations by Mutual Information Estimation and Maximization

Learning Deep Representations by Mutual Information Estimation and Maximization. Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., & Bengio, Y.

Paper abstract bibtex

In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.

@article{hjelmLearningDeepRepresentations2018,
  archivePrefix = {arXiv},
  eprinttype = {arxiv},
  eprint = {1808.06670},
  primaryClass = {cs, stat},
  title = {Learning Deep Representations by Mutual Information Estimation and Maximization},
  url = {http://arxiv.org/abs/1808.06670},
  abstract = {In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.},
  urldate = {2019-04-01},
  date = {2018-08-20},
  keywords = {Statistics - Machine Learning,Computer Science - Machine Learning},
  author = {Hjelm, R. Devon and Fedorov, Alex and Lavoie-Marchildon, Samuel and Grewal, Karan and Bachman, Phil and Trischler, Adam and Bengio, Yoshua},
  file = {/home/dimitri/Nextcloud/Zotero/storage/36AV9H8R/Hjelm et al. - 2018 - Learning deep representations by mutual informatio.pdf;/home/dimitri/Nextcloud/Zotero/storage/IXX3GXTN/1808.html}
}

Downloads: 0

{"_id":"4fYkaJbKAgww9RHpP","bibbaseid":"hjelm-fedorov-lavoiemarchildon-grewal-bachman-trischler-bengio-learningdeeprepresentationsbymutualinformationestimationandmaximization","authorIDs":[],"author_short":["Hjelm, R. D.","Fedorov, A.","Lavoie-Marchildon, S.","Grewal, K.","Bachman, P.","Trischler, A.","Bengio, Y."],"bibdata":{"bibtype":"article","type":"article","archiveprefix":"arXiv","eprinttype":"arxiv","eprint":"1808.06670","primaryclass":"cs, stat","title":"Learning Deep Representations by Mutual Information Estimation and Maximization","url":"http://arxiv.org/abs/1808.06670","abstract":"In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.","urldate":"2019-04-01","date":"2018-08-20","keywords":"Statistics - Machine Learning,Computer Science - Machine Learning","author":[{"propositions":[],"lastnames":["Hjelm"],"firstnames":["R.","Devon"],"suffixes":[]},{"propositions":[],"lastnames":["Fedorov"],"firstnames":["Alex"],"suffixes":[]},{"propositions":[],"lastnames":["Lavoie-Marchildon"],"firstnames":["Samuel"],"suffixes":[]},{"propositions":[],"lastnames":["Grewal"],"firstnames":["Karan"],"suffixes":[]},{"propositions":[],"lastnames":["Bachman"],"firstnames":["Phil"],"suffixes":[]},{"propositions":[],"lastnames":["Trischler"],"firstnames":["Adam"],"suffixes":[]},{"propositions":[],"lastnames":["Bengio"],"firstnames":["Yoshua"],"suffixes":[]}],"file":"/home/dimitri/Nextcloud/Zotero/storage/36AV9H8R/Hjelm et al. - 2018 - Learning deep representations by mutual informatio.pdf;/home/dimitri/Nextcloud/Zotero/storage/IXX3GXTN/1808.html","bibtex":"@article{hjelmLearningDeepRepresentations2018,\n archivePrefix = {arXiv},\n eprinttype = {arxiv},\n eprint = {1808.06670},\n primaryClass = {cs, stat},\n title = {Learning Deep Representations by Mutual Information Estimation and Maximization},\n url = {http://arxiv.org/abs/1808.06670},\n abstract = {In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.},\n urldate = {2019-04-01},\n date = {2018-08-20},\n keywords = {Statistics - Machine Learning,Computer Science - Machine Learning},\n author = {Hjelm, R. Devon and Fedorov, Alex and Lavoie-Marchildon, Samuel and Grewal, Karan and Bachman, Phil and Trischler, Adam and Bengio, Yoshua},\n file = {/home/dimitri/Nextcloud/Zotero/storage/36AV9H8R/Hjelm et al. - 2018 - Learning deep representations by mutual informatio.pdf;/home/dimitri/Nextcloud/Zotero/storage/IXX3GXTN/1808.html}\n}\n\n","author_short":["Hjelm, R. D.","Fedorov, A.","Lavoie-Marchildon, S.","Grewal, K.","Bachman, P.","Trischler, A.","Bengio, Y."],"key":"hjelmLearningDeepRepresentations2018","id":"hjelmLearningDeepRepresentations2018","bibbaseid":"hjelm-fedorov-lavoiemarchildon-grewal-bachman-trischler-bengio-learningdeeprepresentationsbymutualinformationestimationandmaximization","role":"author","urls":{"Paper":"http://arxiv.org/abs/1808.06670"},"keyword":["Statistics - Machine Learning","Computer Science - Machine Learning"],"downloads":0},"bibtype":"article","biburl":"https://raw.githubusercontent.com/dlozeve/newblog/master/bib/all.bib","creationDate":"2020-01-08T20:39:39.312Z","downloads":0,"keywords":["statistics - machine learning","computer science - machine learning"],"search_terms":["learning","deep","representations","mutual","information","estimation","maximization","hjelm","fedorov","lavoie-marchildon","grewal","bachman","trischler","bengio"],"title":"Learning Deep Representations by Mutual Information Estimation and Maximization","year":null,"dataSources":["3XqdvqRE7zuX4cm8m"]}