Pseudo-MDPs and Factored Linear Action Models

Pseudo-MDPs and Factored Linear Action Models. Yao, H., Szepesvári, C., Pires, B., & Zhang, X. In IEEE ADPRL, pages 189–197, 10, 2014.

Paper abstract bibtex 3 downloads

In this paper we introduce the concept of pseudo-MDPs to develop abstractions. Pseudo-MDPs relax the requirement that the transition kernel has to be a probability kernel. We show that the new framework captures many existing abstractions. We also introduce the concept of factored linear action models; a special case. Again, the relation of factored linear action models and existing works are discussed. We use the general framework to develop a theory for bounding the suboptimality of policies derived from pseudo-MDPs. Specializing the framework, we recover existing results. We give a least-squares approach and a constrained optimization approach of learning the factored linear model as well as efficient computation methods. We demonstrate that the constrained optimization approach gives better performance than the least-squares approach with normalization.

@inproceedings{YaoSze14,
	abstract = {In this paper we introduce the concept of pseudo-MDPs
to develop abstractions.
Pseudo-MDPs relax the requirement that the transition kernel has to be a probability kernel.
We show that the new framework captures many existing abstractions.
We also introduce the concept of factored linear action models; a special case.
Again, the relation of factored linear action models and existing works are discussed.
We use the general framework to develop a theory for bounding the suboptimality of policies derived from pseudo-MDPs.
Specializing the framework, we recover existing results.
We give a least-squares approach and a constrained optimization approach of learning the factored linear model as well as efficient computation methods.
We demonstrate that the constrained optimization approach gives better performance than the least-squares approach with normalization.
},
	author = {Yao, H. and Szepesv{\'a}ri, Cs. and Pires, B.A. and Zhang, X.},
	booktitle = {IEEE ADPRL},
	keywords = {factored linear models, reinforcement learning, Markov Decision Processes, function approximation, control, planning, control learning, abstraction, model-based RL, pseudo-MDPs},
	month = {10},
	pages = {189--197},
	title = {Pseudo-MDPs and Factored Linear Action Models},
	url_paper = {ieee_adprl2014.pdf},
	year = {2014}}

Downloads: 3