In this paper, staff researchers from the Bank of England use novel data and machine learning techniques to build an early warning system for UK bank distress. They compare a number of machine learning and classical statistical techniques, implementing a rigorous, double-block randomized cross-validation procedure to evaluate out-of-sample performance.
Researchers find the random forest algorithm to be superior in terms of ranking test observations, while also having relatively better calibrated probabilities than the other techniques. They also examine performance at two different decision thresholds, 50% and 25%, and vary the relative cost of misclassification between FN and FP errors, demonstrating random forest to have lower cost as the weight changes in favour of the former over the latter.
The performance results indicate that the random forest should be used to build an early warning system. In order to improve the algorithm’s transparency, they examine the drivers of the model’s predicted probabilities using Shapley regression, which revealed the importance of macroeconomic variables (especially year-on-year change of average UK real earnings), and a firm’s sensitivity to market risk (ratio of trading book to total assets), capital buffer and net interest margin.
Finally, researchers also perform simple ensembling techniques to combine all the model outputs, demonstrating substantive and statistically significant improvements relative to the random forest on its own.
Future research might extend the analysis in several ways, noted the researchers, including the use of additional data beyond financial ratios and macroeconomic variables. For example, textual data which sheds light on the quality of a firm’s management and governance, or metrics which capture aspects of a firm’s cultures would enrich the set of input variables.
Overall, this research paper demonstrates the practical benefits of machine learning and ensembling methods for providing regulators with advance warning of firm distress. Supervisors can apply these findings in order to aid in anticipating problems before they occur, thereby helping them in their mission to keep financial institutions safe and sound.