You are here:
NEW CATEGORICAL METRICS FOR AIR QUALITY MODEL EVALUATION
Citation:
KANG, D., R. MATHUR, K. L. SCHERE, S. YU, AND B. K. EDER. NEW CATEGORICAL METRICS FOR AIR QUALITY MODEL EVALUATION. JOURNAL OF APPLIED METEOROLOGY AND CLIMATOLOGY. American Meteorological Society, Boston, MA, 46(4):549-555, (2007).
Impact/Purpose:
The objectives of this task include: (1) to continuously evaluate and analyze the forecast results to provide diagnostic information on model performance and inadequacies to guide further evolution and refinements to the CMAQ model, and (2) extending the utility of the daily air quality forecast model data being produced by NOAA's National Weather Service (NWS) as part of a NOAA/EPA collaboration in air quality forecasting, to EPA mission-oriented activities. These objectives include developing and maintaining a long-term database of air quality modeling results (ozone and PM2.5), performing periodic analysis and assessments using the data, and making the air quality database available and accessible to States, Regions, RPO's and others to use as input data for regional/local scale air quality modeling for policy/regulatory purposes.
Description:
Traditional categorical metrics used in model evaluations are "clear-cut" measures in that the model's ability to predict an exceedance is defined by a fixed threshold concentration and the metrics are defined by observation-forecast sets that are paired both in space and time. These metrics are informative but limited in evaluating the performance of air quality forecast (AQF) systems because AQF generally examines exceedances on a regional scale rather than a single monitor.
New categorical metrics, Weighted Success Index (WSI), Area Hit (aH), and Area False Alarm Ratio (aFAR) are developed. In the calculation of WSI, credits are given to the observation-forecast pairs within the observed exceedance region (missed forecast) or the forecast exceedance region (false alarm) depending on the distance of the points from the central line (perfect observation-forecast match line - 1:1 line on scatter plot).The aH and aFAR are defined by matching observed and forecast exceedances within an area (i.e., model grid cells) surrounding the observation location. The concept of aH and aFAR resembles the manner in which forecasts are usually issued. In practice, a warning is issued for a region of interest such as a metropolitan area if an exceedance is forecast to occur anywhere within the region. The application of these new categorical metrics which are supplemental to the traditional counterparts (critical success index, hit rate, and false alarm ratio) to the Eta-CMAQ forecast system has demonstrated further insight into evaluating the forecasting capability of the system (for example, the new metrics can provide the information about how the AQF system captures the regional signals).