Citation:

Hill, R., EricW Fox, S. Leibowitz, Tony Olsen, M. Weber, AND D. Thornbrugh. Variable selection with random forest: Balancing stability, performance, and interpretation in ecological and environmental modeling. Annual Meeting of the Society for Freshwater Science, Sacramento, CA, May 21 - 26, 2016.

Impact/Purpose:

This study supports the development of a robust national map of stream condition for perennial streams, which is of interest to the Monitoring Branch within the Office of Water. We used random forest (RF) to produce this map of national condition. RF is a popular tool in ecological and environmental modeling and it has been suggested that RF models can be improved by reducing the number of variables in a final model through a selection process. Understanding how reducing variables affects final predictions produced by RF is critical to producing a robust national map of stream condition. We examined the effect of variable selection on RF models and will provide suggestions for how and when variable selection should be used with RF modeling. The map produced by this work will also be used to investigate national patterns in health and economic benefits. It also contributes to an FY16 deliverable under SSWR 3.01B.

Description:

Random forest (RF) is popular in ecological and environmental modeling, in part, because of its insensitivity to correlated predictors and resistance to overfitting. Although variable selection has been proposed to improve both performance and interpretation of RF models, it is uncertain how selection affects final predictions. We used forward and backward variable selection on 212 landscape predictors from the EPA’s StreamCat Dataset (anthropogenic and natural metrics) to produce four models of benthic condition (good vs. poor condition as a binary response). Variable selection produced models with 10-15 predictors, and evaluations suggested excellent performances (e.g., AUC = 0.82-0.86, ≈78% correct classification). Selection improved AUC values by up to 5 points compared to the 212-predictor model. Despite similar performances, correspondence of predicted probabilities among models varied greatly (r2 = 0.55-0.75) and produced markedly different maps. Moreover, removal or addition of predictors to these reduced-set models substantially altered predicted values. This instability reduced both confidence and interpretability in candidate models; therefore, we suggest the use of all predictors or a set of uncorrelated predictors in RF modeling with minimal selection.

Record Details: