Diagnosing and Comparing Sources of Error in Emulators of Earth System Models

Abstract

Climate emulators play an increasingly important role in reconciling the computational costs of high-fidelity Earth system models with the need for large ensembles of projections required for robust climate impact assessment. However, the growing number of available emulation techniques makes direct comparisons of their relative strengths and weaknesses difficult. Here, we present a framework to diagnose and compare sources of error across disparate emulation techniques, focusing on common challenges such as memory effects, hidden variables, system noise, and nonlinearities. We compare popular techniques like pattern scaling and response functions, as well as less common methods such as Dynamic Mode Decomposition and the Fluctuation Dissipation Theorem (FDT). We evaluate the performance of these techniques across a series of experiments, each designed to isolate and highlight a different potential source of error. We find that response function-based emulators outperform other techniques, particularly pattern scaling, across all scenarios tested. Results highlight the relative utility of each technique discussed, along with the importance of designing future scenarios for Earth system models with emulation in mind. We additionally outline the potential advantages of incorporating knowledge of the underlying system dynamics into climate emulation, showing that emulation based on the FDT approximates the true system response. Although it requires non-standard training scenarios, this approach allows for interpretable, physics-based emulation, which is critical for building confidence in emulator projections.

Plain-language Summary

Understanding future climate risks requires a large number of projections for many different scenarios. Climate emulators allow for rapid projections without the computational costs associated with full scale climate models. However, many different types of emulators now exist, making it difficult to determine which techniques are best suited for a given task. Here, we outline a framework to compare these different techniques both theoretically and practically through a series of stress tests to pinpoint common sources of emulator error. One popular approach, response functions, consistently outperforms other techniques across all experiments. Our results help clarify which emulators are best suited for different tasks and show how future climate models can be used to support emulator design. Finally, we highlight the advantages of using physically-interpretable emulators to build confidence in emulator projections as they are incorporated into decision-making processes.