Revisiting Kappa to account for change in the accuracy assessment of land-use change models. van Vliet, J., Bregt, A. K., & Hagen-Zanker, A. Ecological Modelling, 222(8):1367–1375, April, 2011.
Revisiting Kappa to account for change in the accuracy assessment of land-use change models [link]Paper  doi  abstract   bibtex   
Land-use change models are typically calibrated to reproduce known historic changes. Calibration results can then be assessed by comparing two datasets: the simulated land-use map and the actual land-use map at the same time. A common method for this is the Kappa statistic, which expresses the agreement between two categorical datasets corrected for the expected agreement. This expected agreement is based on a stochastic model of random allocation given the distribution of class sizes. However, when a model starts from an initial land-use map and makes changes to it, that stochastic model does not pose a meaningful reference level. This paper introduces KSimulation, a statistic that is identical in form to the Kappa statistic but instead applies a more appropriate stochastic model of random allocation of class transitions relative to the initial map. The new method is illustrated on a simple example and then the results of the Kappa statistic and KSimulation are compared using the results of a land-use model. It is found that only KSimulation truly tests models in their capacity to explain land-use changes over time, and unlike Kappa it does not inflate results for simulations where little change takes place over time.
@article{van_vliet_revisiting_2011,
	title = {Revisiting {Kappa} to account for change in the accuracy assessment of land-use change models},
	volume = {222},
	issn = {0304-3800},
	url = {http://www.sciencedirect.com/science/article/pii/S0304380011000494},
	doi = {10.1016/j.ecolmodel.2011.01.017},
	abstract = {Land-use change models are typically calibrated to reproduce known historic changes. Calibration results can then be assessed by comparing two datasets: the simulated land-use map and the actual land-use map at the same time. A common method for this is the Kappa statistic, which expresses the agreement between two categorical datasets corrected for the expected agreement. This expected agreement is based on a stochastic model of random allocation given the distribution of class sizes. However, when a model starts from an initial land-use map and makes changes to it, that stochastic model does not pose a meaningful reference level. This paper introduces KSimulation, a statistic that is identical in form to the Kappa statistic but instead applies a more appropriate stochastic model of random allocation of class transitions relative to the initial map. The new method is illustrated on a simple example and then the results of the Kappa statistic and KSimulation are compared using the results of a land-use model. It is found that only KSimulation truly tests models in their capacity to explain land-use changes over time, and unlike Kappa it does not inflate results for simulations where little change takes place over time.},
	number = {8},
	urldate = {2014-01-10TZ},
	journal = {Ecological Modelling},
	author = {van Vliet, J. and Bregt, A. K. and Hagen-Zanker, A.},
	month = apr,
	year = {2011},
	keywords = {Accuracy assessment, Kappa statistic, Land-use change, Model calibration, map comparison},
	pages = {1367--1375}
}

Downloads: 0