Does the Interpolation Accuracy of Species Distribution Models Come at the Expense of Transferability?. Heikkinen, R. K., Marmion, M., & Luoto, M. Ecography, 35(3):276–288, March, 2012.
doi  abstract   bibtex   
Model transferability (extrapolative accuracy) is one important feature in species distribution models, required in several ecological and conservation biological applications. This study uses 10 modelling techniques and nationwide data on both (1) species distribution of birds, butterflies, and plants and (2) climate and land cover in Finland to investigate whether good interpolative prediction accuracy for models comes at the expense of transferability - i.e. markedly worse performance in new areas. Models' interpolation and extrapolation performance was primarily assessed using AUC (the area under the curve of a receiver characteristic plot) and Kappa statistics, with supplementary comparisons examining model sensitivity and specificity values. Our AUC and Kappa results show that extrapolation to new areas is a greater challenge for all included modelling techniques than simple filling of gaps in a well-sampled area, but there are also differences among the techniques in the degree of transferability. Among the machine-learning modelling techniques, MAXENT, generalized boosting methods (GBM), and artificial neural networks (ANN) showed good transferability while the performance of GARP and random forest (RF) decreased notably in extrapolation. Among the regression-based methods, generalized additive models (GAM) and generalized linear models (GLM) showed good transferability. A desirable combination of good prediction accuracy and good transferability was evident for three modelling techniques: MAXENT, GBM, and GAM. However, examination of model sensitivity and specificity revealed that model types may differ in their tendencies to either increased over-prediction of presences or absences in extrapolation, and some of the methods show contrasting changes in sensitivity vs specificity (e.g. ANN and GARP). Among the three species groups, the best transferability was seen with birds, followed closely by butterflies, whereas reliable extrapolation for plant species distribution models appears to be a major challenge at least at this scale. Overall, detailed knowledge of the behaviour of different techniques in various study settings and with different species groups is of utmost importance in predictive modelling. [Excerpt] [...] For each species, SDMs were generated using ten different algorithms. These included three regression methods [generalized additive models (GAM), generalized linear models (GLM) and multiple adaptive regression splines (MARS)], five machine-learning methods [maximum entropy (MAXENT), artificial neural networks (ANN), generalized boosting method (GBM, also known as boosted regression trees/BRT), random forest (RF) and genetic algorithm for rule set production (GARP)], and two classification methods [classification tree analysis (CTA) and mixture discriminant analysis (MDA)]. [\n] These 10 model types have been described in detail by Stockwell and Peters (1999), Phillips et al. (2006), Olden et al. (2008), and Marmion et al. (2009). All of the model types except MAXENT and GARP require species presence/absence data for model calibration and were applied via the BIOMOD framework (Thuiller et al. 2009). MAXENT and GARP require presence-only species data and were applied via the freely released MaxEnt software (Phillips et al. 2006) and the desktop GARP application (Stockwell and Peters 1999, Pearson et al. 2007). [...] [\n] Among the five machine-learning methods, ANN and MAXENT showed good transferability as indicated by AUC values, GBM and GARP intermediate, but RF the second worst transferability. Thus our results are in agreement with those of Elith et al. (2006), Guisan et al. (2007), and Elith and Graham (2009) where the good performance of MAXENT and GBM is concerned, which apparently includes also good transferability (Duncan et al. 2009, and this study). However, they run counter to the conclusions of a number of recent papers (Lawler et al. 2006, Prasad et al. 2006, Cutler et al. 2007, Syphard and Franklin 2009) with relation to the superiority of RF, especially where model transferability is concerned. It appears that, although RF is able to provide, on average, more accurate predictions within a study landscape than other methods, considerable caution is required if the method is to be used in an extrapolative manner. [\n] Among the five simple model types, both GAM and GLM showed a good - and, especially, MARS a poor - level of transferability. Some earlier studies suggested that MARS can perform equally well as other regression techniques (Leathwick et al. 2006, Guisan et al. 2007). However, our results suggest that this probably holds only in interpolative tasks. Similarly, Prasad et al. (2006) found that MARS was adequate for predicting current species distributions but yielded unacceptable projections for future climates. [...] [::Conclusions] Our results corroborate that transferring of models into new areas is a bigger challenge for many species distribution modelling techniques than simple filling of gaps in a generally sampled landscape. However, extrapolation capabilities appear to vary among modelling techniques, in some respects in unexpected ways. Based on our AUC and Kappa results, not all of the novel modelling techniques show good transferability; while MAXENT, GBM, and ANN do, GARP and RF may have only intermediate or modest transferability. Differences are evident also among regression-based methods, where GAM and GLM have good extrapolation capability whereas MARS fails to transfer reliably. Accordingly, there is no simple answer to the question of whether the precision of the novel machine-learning methods comes at the expense of generalisability and transferability (Araújo and Rahbek 2006). The most desirable combination as regards model performance would be good prediction accuracy and good transferability, which was here achieved by MAXENT, GBM and GAM. With regards to model sensitivity and specificity, our results show that different model types may have differing tendencies for increased over-prediction of presences or absences in extrapolation; some of the methods show contrasting changes in sensitivity vs specificity (e.g. ANN and GARP) while others show broadly similar trends (e.g. GAM and GLM). Such differences need to be taken into account in predictive extrapolative modelling so that the most appropriate modelling approaches can be selected based on study aims. [\n] Of the three species groups studied, vascular plants showed lower model transferability, on average, than did birds and butterflies at 10-km resolution, though individual poorly extrapolated models emerged in all three groups. Thus, transferring broad-scale SDMs for vascular plants can apparently be done only with great caution. Overall, the two aspects of model performance, accuracy in interpolation and transferability, appear to be in some respects similarly and in others divergently affected by the differences in algorithms and between species groups. We conclude that detailed knowledge of the behaviour of different techniques in different study settings and with different species is of utmost importance in extrapolative predictive modelling.
@article{heikkinenDoesInterpolationAccuracy2012,
  title = {Does the Interpolation Accuracy of Species Distribution Models Come at the Expense of Transferability?},
  author = {Heikkinen, Risto K. and Marmion, Mathieu and Luoto, Miska},
  year = {2012},
  month = mar,
  volume = {35},
  pages = {276--288},
  issn = {0906-7590},
  doi = {10.1111/j.1600-0587.2011.06999.x},
  abstract = {Model transferability (extrapolative accuracy) is one important feature in species distribution models, required in several ecological and conservation biological applications. This study uses 10 modelling techniques and nationwide data on both (1) species distribution of birds, butterflies, and plants and (2) climate and land cover in Finland to investigate whether good interpolative prediction accuracy for models comes at the expense of transferability - i.e. markedly worse performance in new areas. Models' interpolation and extrapolation performance was primarily assessed using AUC (the area under the curve of a receiver characteristic plot) and Kappa statistics, with supplementary comparisons examining model sensitivity and specificity values. Our AUC and Kappa results show that extrapolation to new areas is a greater challenge for all included modelling techniques than simple filling of gaps in a well-sampled area, but there are also differences among the techniques in the degree of transferability. Among the machine-learning modelling techniques, MAXENT, generalized boosting methods (GBM), and artificial neural networks (ANN) showed good transferability while the performance of GARP and random forest (RF) decreased notably in extrapolation. Among the regression-based methods, generalized additive models (GAM) and generalized linear models (GLM) showed good transferability. A desirable combination of good prediction accuracy and good transferability was evident for three modelling techniques: MAXENT, GBM, and GAM. However, examination of model sensitivity and specificity revealed that model types may differ in their tendencies to either increased over-prediction of presences or absences in extrapolation, and some of the methods show contrasting changes in sensitivity vs specificity (e.g. ANN and GARP). Among the three species groups, the best transferability was seen with birds, followed closely by butterflies, whereas reliable extrapolation for plant species distribution models appears to be a major challenge at least at this scale. Overall, detailed knowledge of the behaviour of different techniques in various study settings and with different species groups is of utmost importance in predictive modelling.

[Excerpt] [...] For each species, SDMs were generated using ten different algorithms. These included three regression methods [generalized additive models (GAM), generalized linear models (GLM) and multiple adaptive regression splines (MARS)], five machine-learning methods [maximum entropy (MAXENT), artificial neural networks (ANN), generalized boosting method (GBM, also known as boosted regression trees/BRT), random forest (RF) and genetic algorithm for rule set production (GARP)], and two classification methods [classification tree analysis (CTA) and mixture discriminant analysis (MDA)]. 

[\textbackslash n] These 10 model types have been described in detail by Stockwell and Peters (1999), Phillips et al. (2006), Olden et al. (2008), and Marmion et al. (2009). All of the model types except MAXENT and GARP require species presence/absence data for model calibration and were applied via the BIOMOD framework (Thuiller et al. 2009). MAXENT and GARP require presence-only species data and were applied via the freely released MaxEnt software (Phillips et al. 2006) and the desktop GARP application (Stockwell and Peters 1999, Pearson et al. 2007). [...]

[\textbackslash n] Among the five machine-learning methods, ANN and MAXENT showed good transferability as indicated by AUC values, GBM and GARP intermediate, but RF the second worst transferability. Thus our results are in agreement with those of Elith et al. (2006), Guisan et al. (2007), and Elith and Graham (2009) where the good performance of MAXENT and GBM is concerned, which apparently includes also good transferability (Duncan et al. 2009, and this study). However, they run counter to the conclusions of a number of recent papers (Lawler et al. 2006, Prasad et al. 2006, Cutler et al. 2007, Syphard and Franklin 2009) with relation to the superiority of RF, especially where model transferability is concerned. It appears that, although RF is able to provide, on average, more accurate predictions within a study landscape than other methods, considerable caution is required if the method is to be used in an extrapolative manner. 

[\textbackslash n] Among the five simple model types, both GAM and GLM showed a good - and, especially, MARS a poor - level of transferability. Some earlier studies suggested that MARS can perform equally well as other regression techniques (Leathwick et al. 2006, Guisan et al. 2007). However, our results suggest that this probably holds only in interpolative tasks. Similarly, Prasad et al. (2006) found that MARS was adequate for predicting current species distributions but yielded unacceptable projections for future climates. [...]

[::Conclusions] Our results corroborate that transferring of models into new areas is a bigger challenge for many species distribution modelling techniques than simple filling of gaps in a generally sampled landscape. However, extrapolation capabilities appear to vary among modelling techniques, in some respects in unexpected ways. Based on our AUC and Kappa results, not all of the novel modelling techniques show good transferability; while MAXENT, GBM, and ANN do, GARP and RF may have only intermediate or modest transferability. Differences are evident also among regression-based methods, where GAM and GLM have good extrapolation capability whereas MARS fails to transfer reliably. Accordingly, there is no simple answer to the question of whether the precision of the novel machine-learning methods comes at the expense of generalisability and transferability (Ara\'ujo and Rahbek 2006). The most desirable combination as regards model performance would be good prediction accuracy and good transferability, which was here achieved by MAXENT, GBM and GAM. With regards to model sensitivity and specificity, our results show that different model types may have differing tendencies for increased over-prediction of presences or absences in extrapolation; some of the methods show contrasting changes in sensitivity vs specificity (e.g. ANN and GARP) while others show broadly similar trends (e.g. GAM and GLM). Such differences need to be taken into account in predictive extrapolative modelling so that the most appropriate modelling approaches can be selected based on study aims.

[\textbackslash n] Of the three species groups studied, vascular plants showed lower model transferability, on average, than did birds and butterflies at 10-km resolution, though individual poorly extrapolated models emerged in all three groups. Thus, transferring broad-scale SDMs for vascular plants can apparently be done only with great caution. Overall, the two aspects of model performance, accuracy in interpolation and transferability, appear to be in some respects similarly and in others divergently affected by the differences in algorithms and between species groups. We conclude that detailed knowledge of the behaviour of different techniques in different study settings and with different species is of utmost importance in extrapolative predictive modelling.},
  journal = {Ecography},
  keywords = {*imported-from-citeulike-INRMM,~INRMM-MiD:c-13559333,artificial-neural-networks,extrapolation-error,generalized-additive-models,habitat-suitability,machine-learning,maxent,model-comparison,modelling,multiple-adaptive-regression-splines,random-forest,regression,species-distribution},
  lccn = {INRMM-MiD:c-13559333},
  number = {3}
}
Downloads: 0
About
Documentation
Keywords
Search
Users
Terms and Conditions of Use
Privacy Policy
Sitemap
© BibBase.org 2021