A Finite-Sample Generalization Bound for Semiparametric Regression: Partially Linear Models

A Finite-Sample Generalization Bound for Semiparametric Regression: Partially Linear Models. Huang, R. & Szepesvári, C. In AISTATS, pages 402–410, 02, 2014.

Paper abstract bibtex

In this paper we provide generalization bounds for semiparametric regression with the so-called partially linear models where the regression function is written as the sum of a linear parametric and a nonlinear, non- parametric function, the latter taken from a some set H with finite entropy-integral. The problem is technically challenging because the parametric part is unconstrained and the model is underdetermined, while the response is allowed to be unbounded with subgaussian tails. Under natural regularity conditions, we bound the generalization error as a function of the Rademacher complexity of H and that of the linear model. Our main tool is a ratio-type concentration inequality for increments of empirical processes, based on which we are able to give an exponential tail bound on the size of the parametric component. We also provide a comparison to alternatives of this technique and discuss why and when the un-constrained parametric part in the model may cause a problem in terms of the expected risk. We also explain by means of a specific example why this problem cannot be detected using the results of classical asymptotic analysis often seen in the statistics literature.

@inproceedings{HuSze14,
	abstract = {In this paper we provide generalization bounds for semiparametric regression with the so-called partially linear models where the regression function is written as the sum of a linear parametric and a nonlinear, non- parametric function, the latter taken from a some set H with finite entropy-integral. The problem is technically challenging because the parametric part is unconstrained and the model is underdetermined, while the response is allowed to be unbounded with subgaussian tails. Under natural regularity conditions, we bound the generalization error as a function of the Rademacher complexity of H and that of the linear model. Our main tool is a ratio-type concentration inequality for increments of empirical processes, based on which we are able to give an exponential tail bound on the size of the parametric component. We also provide a comparison to alternatives of this technique and discuss why and when the un-constrained parametric part in the model may cause a problem in terms of the expected risk. We also explain by means of a specific example why this problem cannot be detected using the results of classical asymptotic analysis often seen in the statistics literature.},
	author = {Huang, R. and Szepesv{\'a}ri, Cs.},
	booktitle = {AISTATS},
	keywords = {nonparametrics, least-squares methods, rate of convergence},
	month = {02},
	pages = {402--410},
	title = {A Finite-Sample Generalization Bound for Semiparametric Regression: Partially Linear Models},
	url_paper = {semiparametric-aistat2014.pdf},
	year = {2014}}

Downloads: 0