Asymptotic Study of Ensemble Methods for Imbalanced Classification

Overview & Synopsis

This thesis is dedicated to the asymptotic study of the ensemble methods, with a particular focus on infinite random forests and bagged nearest neighbors trained on subsamples drawn without replacement, within the framework of binary classification. In the first part of the thesis, we establish a Central Limit Theorem (CLT) for infinite random forests trained on such subsamples. In the setting of imbalanced classification, these estimators suffer from poor representation of the minority class in the training data. To address this issue, we propose a resampling strategy to rebalance the class distribution. While the resulting estimator satisfies a CLT, it remains not consistent. To mitigate this bias, we introduce a debiasing procedure based on the odds ratio, and demonstrate that the resulting estimator is consistent and still satisfies a CLT. We apply our theoretical results to bagged nearest neighbors and show that the debiased bagged 1-NN estimator matches the convergence rate of its subsampled counterpart while attaining lower asymptotic variance in most cases. In the second part of the thesis, we apply our theoritical findings to subsampled Infinite Centered Random Forests (ICRF) for which we prove a CLT with explicit convergence rates and constants. Furthermore, we show that the debiased ICRF achieves a variance reduction compared to the standard ICRF trained on the original data. Altogether, our theoretical analysis highlights the benefits of training random forests on rebalanced datasets followed by a debiasing step, in contrast to training on the original data. Finally, we illustrate the empirical validity of these results through a study applied to a real-world dataset from a Danish non-life insurance company in the context of fraud detection.