A Large-Scale Study of Probabilistic Calibration in Neural Network Regression

Accurate estimation of predictive uncertainty is essential for optimal decision making. However, recent works have shown that current neural networks tend to be miscalibrated, sparking interest in different approaches to calibration. In this paper, we conduct a large-scale empirical study of the probabilistic calibration of neural networks on 57 tabular regression datasets. We consider recalibration, conformal and regularization approaches, and investigate the trade-offs they induce on calibration and sharpness of the predictions. Based on kernel density estimation, we design new differentiable recalibration and regularization methods, yielding new insights into the performance of these approaches. Furthermore, we find conditions under which recalibration and conformal prediction are equivalent. Our study is fully reproducible and implemented in a common code base for fair comparison.