Hybrid gradient boosting-based response surface methodology for modeling nonlinear data

Abstract

The paper introduces a hybrid model, combining Gradient Boosting Machines (GBM) and Response Surface Methodology (RSM) to overcome nonlinear data modeling. A four-factor Central Composite Design (CCD) of 30 experimental runs was simulated to determine the strength of the hybrid GB-RSM model. The RSM model was estimated with a second-order polynomial fitted with maximum likelihood estimation, and the GBM model was trained with five-fold cross-validation, a learning rate of 0.05, tree depths of 3-4, and 1500 trees in R. The GB-RSM model was obtained by training GBM using the RSM residuals to model higher-order nonlinearity. RSM was applied to estimate the linear, quadratic, and interaction terms; the residuals were provided to a GBM to model them to capture non-linearities. Preprocessing was done using normalization and K-Nearest Neighbor imputation of the missing data. Root Mean Square error, Mean Absolute error, as well as Coefficient of Determination R 2 were used to measure the model’s performance. It was found that the RSM model had moderate accuracy (RMSE = 0.4751, MAE = 0.2775, R 2 = 0.8209) and GBM had better predictive accuracy (RMSE = 0.1761, MAE = 0.1178, R 2 = 0.9754). The Hybrid GB-RSM model was better than the two, having the best predictive variance (RMSE = 0.0896, MAE = 0.0491, R 2 = 0.9935). The residual diagnostics showed that the GB-RSM residuals were well distributed around zero, which means low bias and low variance with high stability. Combining the advantages of RSM and GBM, the hybrid framework provides better predictive and optimization capabilities, which is why it can be utilized in various data-driven decision-making and optimization.