speiser10 / rfvs-regression

Random forest variable selection benchmark for continuous outcomes

rfvs-regression

Note: The following work titled A Comparison of Random Forest Variable Selection Methods for Regression Modeling of Continuous Outcomes by O’Connell, N.S., Jaeger, B.C., Bullock, G.S., and Speiser, J.L. is currently submitted for peer review publication.

The goal of rfvs-regression is to compare random forest variable selection techniques for continuous outcomes. We compare several methods available through various R packages and user defined functions from published paper appendices. The code for implementing each RF variable selection approach tested can be found in the function “rfvs()” and each of the methods assessed and compared are given in the following table:

Abbreviation in Paper	Publication	R package	Approach	Type of forest method
None	Breiman 2001	ranger	N/A	Axis
Svetnik	Svetnik 2004	Uses party, code from Hapfelmeier	Performance Based	Conditional Inference
Jiang	Jiang 2004	Uses party, code from Hapfelmeier	Performance Based	Conditional Inference
Caret	Kuhn 2008	caret	Performance Based	Axis
Altmann	Altmann 2010	vita	Test Based	Axis
Boruta	Kursa 2010	Boruta	Test Based	Axis
aorsf - Menze	Menze 2011	aorsf	Performance Based	Oblique
RRF	Deng 2013	RRF	Performance Based	Axis
SRC	Ishwaran 2014	randomForestSRC	Performance Based	Axis
VSURF	Genuer 2015	VSURF	Performance Based	Axis
aorsf-Negation	Jaeger 2023	aorsf	Performance Based	Oblique
aorsf- Permutation	Jaeger 2023	aorsf	Performance Based	Oblique
Axis - SFE	NA	ranger	Test Based	Axis
rfvimptest	Hapfelmeier 2023	rfvimptest	Test Based	Conditional Inference

In this benchmarking study, we pulled datasets from OpenML and modeldata following the criteria and steps outlined below:

Datasets included

A total of 59 datasets met criteria and were used in this benchmarking study. Summary characteristics of these datasets are given in the figure below

Benchmarking Study

We used five replications of split sample validation (i.e., Monte-Carlo cross validation) for each dataset to evaluate RF variable selection methods.

1. First, a dataset was split into training (75%) and testing (25%) sets.

2. Second, each variable selection method was applied to the training data, and the variables selected by each method were saved.

3. Third, a standard axis-based RF model using the R package ranger and an oblique RF using the package aorsf were fit on the training data set using variables selected from each method, and R^2 was recorded based on the test data for each replication, method, and dataset.

4. Fourth, methods of variable selection were compared based on computation time, accuracy measured by R^2, and percent variable reduction

note: If any missing values were present in the training or testing data, they were imputed prior to running variable selection methods using the mean and mode for continuous and categorical predictors, respectively, computed in the training data.

Primary Results Table

We provide the results in the table below for R^2 for downstream models fitted in Axis and Oblique RFs, variable percent reduction (higher % reduction implies more variables eliminated on average), and computation time (in seconds).

	R-Squared (Axis)	R-Squared (Oblique)	Variable Percent Reduced	Time (seconds)
Altman
Mean (SD) Median \[IQR\]	0.41 ( 0.017 ) 0.39 \[ 0.183 , 0.626 \]	0.43 (0.018) 0.44 \[0.223, 0.651\]	0.75 ( 0.012 ) 0.82 \[ 0.667 , 0.9 \]	338.02 ( 37.636 ) 52.63 \[ 19.956 , 379.291 \]
aorsf - Menze
Mean (SD) Median \[IQR\]	0.42 ( 0.018 ) 0.42 \[ 0.212 , 0.693 \]	0.45 (0.018) 0.47 \[0.233, 0.724\]	0.61 ( 0.015 ) 0.67 \[ 0.439 , 0.812 \]	17.8 ( 2.427 ) 3.57 \[ 1.198 , 12.561 \]
aorsf - Permutation
Mean (SD) Median \[IQR\]	0.42 ( 0.019 ) 0.4 \[ 0.22 , 0.694 \]	0.45 (0.019) 0.48 \[0.236, 0.724\]	0.56 ( 0.016 ) 0.6 \[ 0.364 , 0.8 \]	42.67 ( 6.464 ) 4.91 \[ 1.145 , 24.119 \]
Boruta
Mean (SD) Median \[IQR\]	0.41 ( 0.019 ) 0.42 \[ 0.208 , 0.677 \]	0.43 (0.018) 0.44 \[0.225, 0.673\]	0.46 ( 0.019 ) 0.45 \[ 0.133 , 0.774 \]	40.58 ( 4.363 ) 8.35 \[ 3.519 , 47.411 \]
CARET
Mean (SD) Median \[IQR\]	0.43 ( 0.019 ) 0.42 \[ 0.23 , 0.694 \]	0.44 (0.019) 0.44 \[0.218, 0.681\]	0.48 ( 0.02 ) 0.5 \[ 0.111 , 0.826 \]	3544.69 ( 580.392 ) 171.95 \[ 52.846 , 1170.799 \]
rfvimptest
Mean (SD) Median \[IQR\]	0.18 ( 0.017 ) 0.08 \[ -0.005 , 0.32 \]	0.21 (0.017) 0.1 \[0, 0.401\]	0.92 ( 0.002 ) 0.93 \[ 0.9 , 0.95 \]	1068.56 ( 151.485 ) 143.83 \[ 35.449 , 563.047 \]
Jiang
Mean (SD) Median \[IQR\]	0.42 ( 0.019 ) 0.42 \[ 0.22 , 0.693 \]	0.44 (0.019) 0.44 \[0.232, 0.72\]	0.65 ( 0.015 ) 0.7 \[ 0.474 , 0.872 \]	455.36 ( 63.503 ) 41.57 \[ 14.373 , 260.241 \]
SRC
Mean (SD) Median \[IQR\]	0.41 ( 0.018 ) 0.39 \[ 0.199 , 0.666 \]	0.41 (0.018) 0.42 \[0.143, 0.634\]	0.35 ( 0.019 ) 0.27 \[ 0 , 0.667 \]	30.08 ( 1.16 ) 20.87 \[ 13.91 , 47.381 \]
aorsf - Negation
Mean (SD) Median \[IQR\]	0.41 ( 0.019 ) 0.4 \[ 0.182 , 0.671 \]	0.44 (0.019) 0.41 \[0.188, 0.726\]	0.53 ( 0.016 ) 0.58 \[ 0.325 , 0.776 \]	44.32 ( 7.204 ) 5.06 \[ 1.166 , 20.333 \]
None
Mean (SD) Median \[IQR\]	0.4 ( 0.017 ) 0.4 \[ 0.182 , 0.621 \]	0.4 (0.017) 0.4 \[0.137, 0.635\]	0 ( 0 ) 0 \[ 0 , 0 \]	0 ( 0 ) 0 \[ 0 , 0 \]
Axis - SFE
Mean (SD) Median \[IQR\]	0.4 ( 0.018 ) 0.39 \[ 0.196 , 0.624 \]	0.4 (0.017) 0.4 \[0.161, 0.633\]	0.13 ( 0.01 ) 0.05 \[ 0 , 0.243 \]	0.37 ( 0.032 ) 0.08 \[ 0.041 , 0.373 \]
RRF
Mean (SD) Median \[IQR\]	0.4 ( 0.017 ) 0.4 \[ 0.17 , 0.623 \]	0.4 (0.017) 0.39 \[0.137, 0.633\]	0.02 ( 0.004 ) 0 \[ 0 , 0 \]	2.43 ( 0.256 ) 0.6 \[ 0.152 , 2.735 \]
Svetnik
Mean (SD) Median \[IQR\]	0.41 ( 0.018 ) 0.37 \[ 0.181 , 0.642 \]	0.43 (0.018) 0.4 \[0.204, 0.68\]	0.69 ( 0.016 ) 0.76 \[ 0.577 , 0.904 \]	1197.21 ( 134.788 ) 245.47 \[ 76.233 , 1317.819 \]
VSURF
Mean (SD) Median \[IQR\]	0.43 ( 0.018 ) 0.41 \[ 0.205 , 0.69 \]	0.43 (0.019) 0.43 \[0.216, 0.714\]	0.76 ( 0.014 ) 0.84 \[ 0.707 , 0.918 \]	256.19 ( 41.346 ) 23.71 \[ 8.851 , 125.959 \]

Primary Results Figure

We present the results of accuracy, by time, and percent reduction in the figure below. K-Means clustering was used to find the cluster of methods that perform best optimally in terms of computation time and accuracy (in the bottom right corner of the figure), with size and color denoting percent reduction.

We observe that for downstream Axis forests fitted in ranger, the methods of Boruta (r package: boruta) and aorsf-Menze (r package: aorsf) perform optimmaly in terms of fast computation time and high-accuracy while preserving good parsimony (good variable percent reduction).

For downstream oblique forests fitted in aorsf, the methods aorsf-Menze and aorsf-Permutation (both found within the aorsf R package) perform best in terms of computation time and accuracy.

Sensitivity to incomplete replication

We note that in several dataset replications, at least one method failed to select a single variable in variable selection. We performed a sensitivity analysis by assessing results only in replications where all methods selected at least one variable (2,464 out of 4,260)

Comparison of Axis vs Oblique fitted downstream RFs

Last but not least, we compare downstream fitted Axis RFs to Oblique RFs.

We find that in terms of median accuracy, downstream oblique fitted forests generally perform slightly better than downstream Axis forests, particularly among the top performing methods of variable selection.