Beyond Average Calibration: A Study of Local and Conditional Reliability for Trustworthy Uncertainty Quantification
Main article
Abstract
Predictive models are increasingly asked not only to make decisions but to report how confident they are, and that confidence is useful only if it is calibrated: stated uncertainty must correspond to the error a model actually makes. A large literature equips practitioners to measure and improve calibration, yet most of it certifies calibration only on average, asking whether predicted confidence matches accuracy across an entire test set. This article argues that average calibration, or consistency, is necessary but far from sufficient, and that the property which matters for trustworthy deployment is adaptivity: calibration that holds locally, conditioned on the input, so that the uncertainty attached to a particular prediction can be believed. We formalise the distinction between marginal, conditional, and individual calibration, assemble the diagnostics each requires, and apply them to several predictive models on equal footing. We show that a model can be almost perfectly calibrated on average while being badly over-confident in some regions of the input space and under-confident in others, that standard aggregate metrics are blind to this, and that popular recalibration methods which repair average calibration leave local miscalibration largely intact. We conclude that reliable uncertainty quantification must be evaluated and enforced conditionally rather than only marginally.
