Citation:

Solazzo, E., C. Hogrefe, AND S. Galmarini. In Search of a New Paradigm for Evaluating Models. Lessons Learned After Three Phases of the Aqmeii Activity. In Proceedings, 18th International Conference on Harmonisation within Atmospheric Dispersion Modelling, Bologna, ITALY, October 09 - 12, 2017. Inderscience Enterprises Limited, Geneva, Switzerland, 1-5, (2017).

Impact/Purpose:

The National Exposure Research Laboratory (NERL) Computational Exposure Division (CED) develops and evaluates data, decision-support tools, and models to be applied to media-specific or receptor-specific problem areas. CED uses modeling-based approaches to characterize exposures, evaluate fate and transport, and support environmental diagnostics/forensics with input from multiple data sources. It also develops media- and receptor-specific models, process models, and decision support tools for use both within and outside of EPA.

Description:

The Air Quality Model Evaluation International Initiative (AQMEII) has been active since 2010 with the aim of building a coordinated international effort on regional scale air quality modelling and evaluation, involving the modelling communities of North America and Europe. Over the years several dozen modelling groups from both continents have applied their modelling systems to common exercises, simulating air quality for a target year and delivered their results to a shared platform with a high level of harmonisation where they were evaluated against an extensive collection of available observations. The third and most recent phase of the activity was initiated in 2014 and is now nearing its completion. Our experience suggests that the widespread practice of scoring the models using aggregate error metrics does not allow a comprehensive understating of error causes, and that the discussion about ‘goodness’ or ‘badness’ of a model based on such a practice can become sterile as it i) does not target the source of the error, ii) does not indicate if the model is doing the right thing for the right reason, and consequently iii) does not provide enough information for model development and improvement. Within AQMEII we have introduced the error apportionment method, where aggregated error metrics are used for time scale analysis and error qualification. Although this methodology provides a much clearer indication of the time scale and the type of model error with respect to conventional operational model evaluation, it still does not permit the unequivocal attribution of errors to specific processes. We therefore argue that evaluation needs to evolve from a practice into a discipline designed to objectively and diagnostically develop and demonstrate viable performance evaluation techniques for regional air quality modelling systems.

URLs/Downloads:

Record Details: