You are here:
Performance metrics for the assessment of satellite data products: an ocean color case study
Citation:
Seegers, B., R. Stumpf, B. Schaeffer, K. Loftin, AND P. Werdell. Performance metrics for the assessment of satellite data products: an ocean color case study. Optics Express. Optical Society of America (OSA) Publishing, Washington, DC, 26(6):7404-7422, (2018).
Impact/Purpose:
The development and refinement of algorithms to derive geophysical variables from satellite measurements of ocean color has been pursued for decades [1]. These data records play a key role in furthering our scientific understanding of the spatial and temporal distributions of marine phytoplankton and other biogeochemical parameters on regional to global scales. Such parameters provide proxy (surrogate) indicators of marine ecosystem health and link to economically important measures, such as fisheries production, water quality, and recreational opportunities [2-3]. In the four decades since the advent of satellite ocean color, the number of algorithms and approaches to produce geophysical data products has increased substantially given improved knowledge of ocean optics, advances in and an increased volume of in situ measurements, improvements in computing power, and open access to satellite data records. Satellite measurements of ocean color now play an important role in scientific Earth system modeling [4-6] and resource managerment decision support [7]. This growing demand for satellite ocean color data products has necessitated the development and expansion of algorithms to accommodate user demands and requirements that span oceans, coastal marine waters, estuaries, lakes, reservoirs, and large rivers. Accomodating this influx of new and enhanced end-user needs subsequently resulted in a growing difficulty in assessing how algorithm refinements ultimately result in any meaningful or constructive improvement in the accuracy and precision of derived satellite data products. This difficulty partly results from the ocean color science community traditionally relying on a small set of statistical tools for algorithm assessment that may not properly provide the appropriate metrics of overall performance. Estimating
Description:
Performance assessment of ocean color satellite data has generally relied on statistical metrics chosen for their common usage and the rationale for selecting certain metrics is infrequently explained. Commonly reported statistics based on mean squared errors, such as the coefficient of determination (r2), root mean square error, and regression slopes, are most appropriate for Gaussian distributions without outliers and, therefore, are often not ideal for ocean color algorithm performance assessment, which is often limited by sample availability. In contrast, metrics based on simple deviations, such as bias and mean absolute error, as well as pair-wise comparisons, often provide more robust and straightforward quantities for evaluating ocean color algorithms with non-Gaussian distributions and outliers. This study uses a SeaWiFS chlorophyll-a validation data set to demonstrate a framework for satellite data product assessment and recommends a multi-metric and user-dependent approach that can be applied within science, modeling, and resource management communities.
