Beyond Aggregate Fidelity: A Comparative Study of Distributional, Downstream-Utility, and Per-Instance Measures for Evaluating Synthetic Time-Series Data
Main article
Abstract
Synthetic time series produced by generative models are now widely used for data augmentation, privacy-preserving data sharing, and the simulation of rare operating conditions, yet their value depends entirely on how faithfully and usefully they stand in for real data. Deciding whether a synthetic dataset is good, however, remains unsettled: a broad and growing family of evaluation measures exists, spanning distributional fidelity, downstream utility, diversity and coverage, per-instance realism, and privacy leakage, but each measure emphasises a different facet of quality, and the most widely used measures are aggregate, computed over a whole set, and post-hoc, requiring access to real data. This article assembles a structured taxonomy of these measures and applies a common panel of them, on equal footing, to several generators of physiological sensor time series. Three findings recur. Measures disagree on which generator is best, so the verdict depends on the measure chosen; distributional fidelity is only weakly related to downstream utility, so a realistic-looking dataset need not be a useful one; and aggregate scores conceal a heavy tail of low-quality individual samples that a single average cannot reveal. We argue that evaluating synthetic time series calls for a multi-faceted, per-instance, and task-aware protocol rather than any single score, and we set out practical guidance for assembling one.
