Bayesian Entropy Estimation for Countable Discrete Distributions. Archer, E., Park, I. M., & Pillow, J. ArXiv e-prints, February, 2013. ![Bayesian Entropy Estimation for Countable Discrete Distributions [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper abstract bibtex We consider the problem of estimating Shannon's entropy $H$ from discrete data, in cases where the number of possible symbols is unknown or even countably infinite. The Pitman-Yor process, a generalization of Dirichlet process, provides a tractable prior distribution over the space of countably infinite discrete distributions, and has found major applications in Bayesian non-parametric statistics and machine learning. Here we show that it also provides a natural family of priors for Bayesian entropy estimation, due to the fact that moments of the induced posterior distribution over $H$ can be computed analytically. We derive formulas for the posterior mean (Bayes' least squares estimate) and variance under Dirichlet and Pitman-Yor process priors. Moreover, we show that a fixed Dirichlet or Pitman-Yor process prior implies a narrow prior distribution over $H$, meaning the prior strongly determines the entropy estimate in the under-sampled regime. We derive a family of continuous mixing measures such that the resulting mixture of Pitman-Yor processes produces an approximately flat prior over $H$. We show that the resulting Pitman-Yor Mixture (PYM) entropy estimator is consistent for a large class of distributions. We explore the theoretical properties of the resulting estimator, and show that it performs well both in simulation and in application to real data.
@ARTICLE{Archer2013a,
author = {Archer, Evan and Park, Il Memming and Pillow, Jonathan},
title = {{Bayes}ian Entropy Estimation for Countable Discrete Distributions},
journal = {ArXiv e-prints},
year = {2013},
month = feb,
abstract = {We consider the problem of estimating Shannon's entropy $H$ from discrete
data, in cases where the number of possible symbols is unknown or
even countably infinite. The {Pitman-Yor} process, a generalization
of Dirichlet process, provides a tractable prior distribution over
the space of countably infinite discrete distributions, and has found
major applications in Bayesian non-parametric statistics and machine
learning. Here we show that it also provides a natural family of
priors for Bayesian entropy estimation, due to the fact that moments
of the induced posterior distribution over $H$ can be computed analytically.
We derive formulas for the posterior mean (Bayes' least squares estimate)
and variance under Dirichlet and {Pitman-Yor} process priors. Moreover,
we show that a fixed Dirichlet or {Pitman-Yor} process prior implies
a narrow prior distribution over $H$, meaning the prior strongly
determines the entropy estimate in the under-sampled regime. We derive
a family of continuous mixing measures such that the resulting mixture
of {Pitman-Yor} processes produces an approximately flat prior over
$H$. We show that the resulting {Pitman-Yor} Mixture ({PYM}) entropy
estimator is consistent for a large class of distributions. We explore
the theoretical properties of the resulting estimator, and show that
it performs well both in simulation and in application to real data.},
archiveprefix = {arXiv},
citeulike-article-id = {12071222},
citeulike-linkout-0 = {http://arxiv.org/abs/1302.0328},
citeulike-linkout-1 = {http://arxiv.org/pdf/1302.0328},
day = {2},
eprint = {1302.0328},
keywords = {bayesian, entropy-estimation, nonparametric-bayes, pitman-yor-process},
posted-at = {2013-02-25 03:18:15},
primaryclass = {cs.IT},
priority = {0},
url = {http://arxiv.org/abs/1302.0328}
}
Downloads: 0
{"_id":"x2Y85iaEgjmyZJcZN","bibbaseid":"archer-park-pillow-bayesianentropyestimationforcountablediscretedistributions-2013","downloads":0,"creationDate":"2016-10-24T21:01:39.556Z","title":"Bayesian Entropy Estimation for Countable Discrete Distributions","author_short":["Archer, E.","Park, I. M.","Pillow, J."],"year":2013,"bibtype":"article","biburl":"https://raw.githubusercontent.com/catniplab/catniplab.github.io/master/catniplab.bib","bibdata":{"bibtype":"article","type":"article","author":[{"propositions":[],"lastnames":["Archer"],"firstnames":["Evan"],"suffixes":[]},{"propositions":[],"lastnames":["Park"],"firstnames":["Il","Memming"],"suffixes":[]},{"propositions":[],"lastnames":["Pillow"],"firstnames":["Jonathan"],"suffixes":[]}],"title":"Bayesian Entropy Estimation for Countable Discrete Distributions","journal":"ArXiv e-prints","year":"2013","month":"February","abstract":"We consider the problem of estimating Shannon's entropy $H$ from discrete data, in cases where the number of possible symbols is unknown or even countably infinite. The Pitman-Yor process, a generalization of Dirichlet process, provides a tractable prior distribution over the space of countably infinite discrete distributions, and has found major applications in Bayesian non-parametric statistics and machine learning. Here we show that it also provides a natural family of priors for Bayesian entropy estimation, due to the fact that moments of the induced posterior distribution over $H$ can be computed analytically. We derive formulas for the posterior mean (Bayes' least squares estimate) and variance under Dirichlet and Pitman-Yor process priors. Moreover, we show that a fixed Dirichlet or Pitman-Yor process prior implies a narrow prior distribution over $H$, meaning the prior strongly determines the entropy estimate in the under-sampled regime. We derive a family of continuous mixing measures such that the resulting mixture of Pitman-Yor processes produces an approximately flat prior over $H$. We show that the resulting Pitman-Yor Mixture (PYM) entropy estimator is consistent for a large class of distributions. We explore the theoretical properties of the resulting estimator, and show that it performs well both in simulation and in application to real data.","archiveprefix":"arXiv","citeulike-article-id":"12071222","citeulike-linkout-0":"http://arxiv.org/abs/1302.0328","citeulike-linkout-1":"http://arxiv.org/pdf/1302.0328","day":"2","eprint":"1302.0328","keywords":"bayesian, entropy-estimation, nonparametric-bayes, pitman-yor-process","posted-at":"2013-02-25 03:18:15","primaryclass":"cs.IT","priority":"0","url":"http://arxiv.org/abs/1302.0328","bibtex":"@ARTICLE{Archer2013a,\n author = {Archer, Evan and Park, Il Memming and Pillow, Jonathan},\n title = {{Bayes}ian Entropy Estimation for Countable Discrete Distributions},\n journal = {ArXiv e-prints},\n year = {2013},\n month = feb,\n abstract = {We consider the problem of estimating Shannon's entropy $H$ from discrete\n\tdata, in cases where the number of possible symbols is unknown or\n\teven countably infinite. The {Pitman-Yor} process, a generalization\n\tof Dirichlet process, provides a tractable prior distribution over\n\tthe space of countably infinite discrete distributions, and has found\n\tmajor applications in Bayesian non-parametric statistics and machine\n\tlearning. Here we show that it also provides a natural family of\n\tpriors for Bayesian entropy estimation, due to the fact that moments\n\tof the induced posterior distribution over $H$ can be computed analytically.\n\tWe derive formulas for the posterior mean (Bayes' least squares estimate)\n\tand variance under Dirichlet and {Pitman-Yor} process priors. Moreover,\n\twe show that a fixed Dirichlet or {Pitman-Yor} process prior implies\n\ta narrow prior distribution over $H$, meaning the prior strongly\n\tdetermines the entropy estimate in the under-sampled regime. We derive\n\ta family of continuous mixing measures such that the resulting mixture\n\tof {Pitman-Yor} processes produces an approximately flat prior over\n\t$H$. We show that the resulting {Pitman-Yor} Mixture ({PYM}) entropy\n\testimator is consistent for a large class of distributions. We explore\n\tthe theoretical properties of the resulting estimator, and show that\n\tit performs well both in simulation and in application to real data.},\n archiveprefix = {arXiv},\n citeulike-article-id = {12071222},\n citeulike-linkout-0 = {http://arxiv.org/abs/1302.0328},\n citeulike-linkout-1 = {http://arxiv.org/pdf/1302.0328},\n day = {2},\n eprint = {1302.0328},\n keywords = {bayesian, entropy-estimation, nonparametric-bayes, pitman-yor-process},\n posted-at = {2013-02-25 03:18:15},\n primaryclass = {cs.IT},\n priority = {0},\n url = {http://arxiv.org/abs/1302.0328}\n}\n\n","author_short":["Archer, E.","Park, I. M.","Pillow, J."],"key":"Archer2013a","id":"Archer2013a","bibbaseid":"archer-park-pillow-bayesianentropyestimationforcountablediscretedistributions-2013","role":"author","urls":{"Paper":"http://arxiv.org/abs/1302.0328"},"keyword":["bayesian","entropy-estimation","nonparametric-bayes","pitman-yor-process"],"downloads":0,"html":""},"search_terms":["bayesian","entropy","estimation","countable","discrete","distributions","archer","park","pillow"],"keywords":["bayesian","entropy-estimation","nonparametric-bayes","pitman-yor-process"],"authorIDs":["580e76b361ea82f22c000017"],"dataSources":["MSNTC753AtSzTtTJ8"]}