A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms

A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms. Bengio, Y., Deleu, T., Rahaman, N., Ke, R., Lachapelle, S., Bilaniuk, O., Goyal, A., & Pal, C. , 2019.
abstract bibtex

We propose to meta-learn causal structures based on how fast a learner adapts to new distributions arising from sparse distributional changes, e.g. due to interventions, actions of agents and other sources of non-stationarities. We show that under this assumption, the correct causal structural choices lead to faster adaptation to modified distributions because the changes are concentrated in one or just a few mechanisms when the learned knowledge is modularized appropriately. This leads to sparse expected gradients and a lower effective number of degrees of freedom needing to be relearned while adapting to the change. It motivates using the speed of adaptation to a modified distribution as a meta-learning objective. We demonstrate how this can be used to determine the cause-effect relationship between two observed variables. The distributional changes do not need to correspond to standard interventions (clamping a variable), and the learner has no direct knowledge of these interventions. We show that causal structures can be parameterized via continuous variables and learned end-to-end. We then explore how these ideas could be used to also learn an encoder that would map low-level observed variables to unobserved causal variables leading to faster adaptation out-of-distribution, learning a representation space where one can satisfy the assumptions of independent mechanisms and of small and sparse changes in these mechanisms due to actions and non-stationarities.

@Article{Bengio2019,
author = {Bengio, Yoshua and Deleu, Tristan and Rahaman, Nasim and Ke, Rosemary and Lachapelle, Sébastien and Bilaniuk, Olexa and Goyal, Anirudh and Pal, Christopher}, 
title = {A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms}, 
journal = {}, 
volume = {}, 
number = {}, 
pages = {}, 
year = {2019}, 
abstract = {We propose to meta-learn causal structures based on how fast a learner adapts to new distributions arising from sparse distributional changes, e.g. due to interventions, actions of agents and other sources of non-stationarities. We show that under this assumption, the correct causal structural choices lead to faster adaptation to modified distributions because the changes are concentrated in one or just a few mechanisms when the learned knowledge is modularized appropriately. This leads to sparse expected gradients and a lower effective number of degrees of freedom needing to be relearned while adapting to the change. It motivates using the speed of adaptation to a modified distribution as a meta-learning objective. We demonstrate how this can be used to determine the cause-effect relationship between two observed variables. The distributional changes do not need to correspond to standard interventions (clamping a variable), and the learner has no direct knowledge of these interventions. We show that causal structures can be parameterized via continuous variables and learned end-to-end. We then explore how these ideas could be used to also learn an encoder that would map low-level observed variables to unobserved causal variables leading to faster adaptation out-of-distribution, learning a representation space where one can satisfy the assumptions of independent mechanisms and of small and sparse changes in these mechanisms due to actions and non-stationarities.}, 
location = {}, 
keywords = {}}

Downloads: 0

{"_id":"jNxWyGfHpxBPSHpRx","bibbaseid":"bengio-deleu-rahaman-ke-lachapelle-bilaniuk-goyal-pal-ametatransferobjectiveforlearningtodisentanglecausalmechanisms-2019","authorIDs":[],"author_short":["Bengio, Y.","Deleu, T.","Rahaman, N.","Ke, R.","Lachapelle, S.","Bilaniuk, O.","Goyal, A.","Pal, C."],"bibdata":{"bibtype":"article","type":"article","author":[{"propositions":[],"lastnames":["Bengio"],"firstnames":["Yoshua"],"suffixes":[]},{"propositions":[],"lastnames":["Deleu"],"firstnames":["Tristan"],"suffixes":[]},{"propositions":[],"lastnames":["Rahaman"],"firstnames":["Nasim"],"suffixes":[]},{"propositions":[],"lastnames":["Ke"],"firstnames":["Rosemary"],"suffixes":[]},{"propositions":[],"lastnames":["Lachapelle"],"firstnames":["Sébastien"],"suffixes":[]},{"propositions":[],"lastnames":["Bilaniuk"],"firstnames":["Olexa"],"suffixes":[]},{"propositions":[],"lastnames":["Goyal"],"firstnames":["Anirudh"],"suffixes":[]},{"propositions":[],"lastnames":["Pal"],"firstnames":["Christopher"],"suffixes":[]}],"title":"A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms","journal":"","volume":"","number":"","pages":"","year":"2019","abstract":"We propose to meta-learn causal structures based on how fast a learner adapts to new distributions arising from sparse distributional changes, e.g. due to interventions, actions of agents and other sources of non-stationarities. We show that under this assumption, the correct causal structural choices lead to faster adaptation to modified distributions because the changes are concentrated in one or just a few mechanisms when the learned knowledge is modularized appropriately. This leads to sparse expected gradients and a lower effective number of degrees of freedom needing to be relearned while adapting to the change. It motivates using the speed of adaptation to a modified distribution as a meta-learning objective. We demonstrate how this can be used to determine the cause-effect relationship between two observed variables. The distributional changes do not need to correspond to standard interventions (clamping a variable), and the learner has no direct knowledge of these interventions. We show that causal structures can be parameterized via continuous variables and learned end-to-end. We then explore how these ideas could be used to also learn an encoder that would map low-level observed variables to unobserved causal variables leading to faster adaptation out-of-distribution, learning a representation space where one can satisfy the assumptions of independent mechanisms and of small and sparse changes in these mechanisms due to actions and non-stationarities.","location":"","keywords":"","bibtex":"@Article{Bengio2019,\nauthor = {Bengio, Yoshua and Deleu, Tristan and Rahaman, Nasim and Ke, Rosemary and Lachapelle, Sébastien and Bilaniuk, Olexa and Goyal, Anirudh and Pal, Christopher}, \ntitle = {A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms}, \njournal = {}, \nvolume = {}, \nnumber = {}, \npages = {}, \nyear = {2019}, \nabstract = {We propose to meta-learn causal structures based on how fast a learner adapts to new distributions arising from sparse distributional changes, e.g. due to interventions, actions of agents and other sources of non-stationarities. We show that under this assumption, the correct causal structural choices lead to faster adaptation to modified distributions because the changes are concentrated in one or just a few mechanisms when the learned knowledge is modularized appropriately. This leads to sparse expected gradients and a lower effective number of degrees of freedom needing to be relearned while adapting to the change. It motivates using the speed of adaptation to a modified distribution as a meta-learning objective. We demonstrate how this can be used to determine the cause-effect relationship between two observed variables. The distributional changes do not need to correspond to standard interventions (clamping a variable), and the learner has no direct knowledge of these interventions. We show that causal structures can be parameterized via continuous variables and learned end-to-end. We then explore how these ideas could be used to also learn an encoder that would map low-level observed variables to unobserved causal variables leading to faster adaptation out-of-distribution, learning a representation space where one can satisfy the assumptions of independent mechanisms and of small and sparse changes in these mechanisms due to actions and non-stationarities.}, \nlocation = {}, \nkeywords = {}}\n\n\n","author_short":["Bengio, Y.","Deleu, T.","Rahaman, N.","Ke, R.","Lachapelle, S.","Bilaniuk, O.","Goyal, A.","Pal, C."],"key":"Bengio2019","id":"Bengio2019","bibbaseid":"bengio-deleu-rahaman-ke-lachapelle-bilaniuk-goyal-pal-ametatransferobjectiveforlearningtodisentanglecausalmechanisms-2019","role":"author","urls":{},"downloads":0},"bibtype":"article","biburl":"https://gist.githubusercontent.com/stuhlmueller/a37ef2ef4f378ebcb73d249fe0f8377a/raw/6f96f6f779501bd9482896af3e4db4de88c35079/references.bib","creationDate":"2020-01-27T02:13:33.887Z","downloads":0,"keywords":[],"search_terms":["meta","transfer","objective","learning","disentangle","causal","mechanisms","bengio","deleu","rahaman","ke","lachapelle","bilaniuk","goyal","pal"],"title":"A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms","year":2019,"dataSources":["hEoKh4ygEAWbAZ5iy"]}