Grantee Research Project Results
Final Report: Linking Environmental and Social Performance Measurement for Management at National and Watershed Levels: Modeling and Statistical Approaches
EPA Grant Number: R828021Title: Linking Environmental and Social Performance Measurement for Management at National and Watershed Levels: Modeling and Statistical Approaches
Investigators: Small, Mitchell J. , Solow, Andrew R. , Fischbeck, Paul S. , Farrow, Scott , Bondelid, Tim , Sinnott, James , Schultz, Martin , Van Houtven, George L. , Schoen, Mary E. , Cooter, W.S.
Institution: Carnegie Mellon University , Desert Research Institute , Woods Hole Oceanographic Institution
EPA Project Officer: Packard, Benjamin H
Project Period: January 10, 2000 through January 9, 2003
Project Amount: $649,864
RFA: Water and Watersheds (1999) RFA Text | Recipients Lists
Research Category: Watersheds , Water
Objective:
The overall objective of this research project was to integrate physical, ecological, and social science models and data to provide an evaluation tool for surface water quality managers at various levels of spatial aggregation. The principal model used in this study is the National Water Pollution Control Assessment Model (NWPCAM), a watershed-based water quality model with national scope. NWPCAM was developed by the Research Triangle Institute, Inc. (RTI) for the U.S. Environmental Protection Agency (EPA) under a previous contract for use in the retrospective evaluation of Clean Water Act Benefits, and has been under continuous development since 1997. NWPCAM has found applications in several of EPA’s regulatory efforts, including a retrospective study of benefits under the Clean Water Act (Bingham, et al., 2000), and an evaluation of proposed regulations to limit discharges from concentrated animal feeding operations (EPA, 2001), the meat and poultry product industry (EPA, 2002), and the metal products and machinery industry (EPA, 2003). Our research involved several layers of effort to test the model, assess its performance, interpret its outputs, and enhance its utility for watershed and environmental managers.
Summary/Accomplishments (Outputs/Outcomes):
Model Summary
Version 1.1 of NWPCAM is a steady-state model that predicts, at approximately 1-mile intervals, five conventional water quality constituents for 632,000 miles of Reach File 1 (RF-1) rivers and streams located throughout the 48-contiguous states. The original version of NWPCAM, summarized in detail in Bondelid, et al. (2000), estimates ambient levels of conventional water quality constituents, including biological oxygen demand (BOD), total Kjeldahl nitrogen (TKN), total suspended solids (TSS), fecal coliform (FC), and dissolved oxygen. Research conducted during the grant period has enabled modifications that incorporate modules for the prediction of nutrients and toxics, including metals (e.g., RTI, 2003; Little, et al., 2004). Estimates of water quality parameters are used to infer the level of use-support (boatable, fishable, or swimmable) in each stream mile. NWPCAM is written in Visual Basic and runs in Microsoft Access.
Model Assessment: Testing and Validation
Model assessment addresses the integration of physical, ecological, and social performance measurement at the watershed level by helping to establish the credibility of the physical and ecological modeling tool. Our assessment consisted of testing to validate operational aspects of the computer program and the predictive ability of the water quality model. Both were significant foci of this research effort. Carnegie Mellon University undertook testing of NWPCAM using a battery of heuristic tests to determine whether the model operated as described in its documentation. The tests revealed numerous issues and potential flaws in the water quality model, the user interface, and the input databases. Specific information regarding these flaws was provided to the project team members responsible for its formulation, and many of these issues have since been addressed through modifications to the model and its input files.
An assessment of NWPCAM’s predictive ability is based on a comparison of model results with field observations from the six watersheds in the Monongahela River Basin, which consists of the entire area upstream of the confluence of the Monongahela River with the Allegheny and Ohio Rivers at Pittsburgh, Pennsylvania. The watershed extends south from Pittsburgh into West Virginia and Maryland and includes the Monongahela River, the Cheat River, the West Fork River, the Tygart River, and the Youghiogheny River watersheds. Predictive ability is tested by determining whether there is a positive and statistically significant correlation among NWPCAM predictions and field observations for each water quality constituent at 92 sampling locations in the sub-basin. Because NWPCAM predicts steady state conditions, correlations also were calculated using long-term mean constituent levels at sampling locations, where possible. The notion of steady state is a useful abstraction for modeling, but cannot be observed directly. A steady state can only be approximated using long-term means.
Comparison of model predictions and water quality observations from the mid-1990s revealed that, with the exception of fecal coliform predictions, NWPCAM estimates were generally within the range of observed levels. Although in range, these predictions were not, however, highly correlated with STORET water quality observations in the watershed. Results also showed that the version of NWPCAM tested at that time under-predicted observed FC concentrations by several orders of magnitude, and this issue has since been addressed through modifications to the model.
Because use support is determined by considering four separate water quality constituents, the determination of economic benefits depends more heavily on the ability to predict levels of, and changes in, use support than on individual water quality constituents. Therefore, predictive ability is assessed in this study by comparing use-support predictions based on information that is available in STORET and the corresponding information from NWPCAM. Categorical measures of association, like correlation coefficients, provide a quantitative measure of how well the model predicts the use-support levels in the watershed. Results of this reliability assessment for use-support classification show a positive and statistically significant correspondence between these classification approaches, although the level of agreement was not high.
Because STORET data are sparse, unevenly distributed in space and time, and collected under a variety of environmental conditions, they represent an imperfect record of the system being modeled and will give an opportunistic character to any benchmarking study based on them. In addition, no single regional assessment can provide a conclusive evaluation of the model’s ability to predict baseline conditions or changes in water quality at a national scale. Even fully adequate benchmarking studies must be replicated in many watersheds to demonstrate conclusively the credibility of a large-scale model like NWPCAM.
Efforts to Reduce Prediction Error
In an effort to reduce prediction error, RTI has pursued the calibration of NWPCAM and has determined that traditional measures of model fit, such as Least Squares (LS), are predicated upon the analyst having pre-screened the observed data for inconsistencies and outliers. For instance, LS can be “driven” by a few outliers. Water quality data are notoriously “noisy,” with observations from low-flow, normal, and storm events, combined with simple data entry problems, including mismatched units. Given the scope of NWPCAM, manual pre-screening of “noisy” data is impractical. As an alternative, RTI has implemented a different measure, called Least Median Square (LMS), as the calibration objective function. This measure has been demonstrated to be particularly effective at separating pattern from noise. In addition, RTI has developed semi-automated calibration procedures to determine model coefficients that best match the observed data from the STORET database. These methods are now being used in initial calibration studies for a new version of NWPCAM designed to estimate nutrient loadings and resulting eutrophication impacts (Little, et al., 2004).
Efforts to Assist Watershed Managers and Address Uncertainty
Watershed managers and regulatory agencies that use NWPCAM to evaluate policy choices face a difficult task incorporating uncertainty into their decision process because NWPCAM’s large output files and runtime requirements make Monte Carlo simulation costly. This is one reason that previous applications of NWPCAM have relied on point estimates with no information about uncertainty. An important result of our research effort has been the development of a reduced-form model (RFM) that can reduce simulation costs and provide watershed managers with capabilities they do not have with the original NWPCAM. The RFM is a single, continuous function that estimates the proportion of stream miles in each use-support category as a function of pollution control variables and explicit physical relationships known from water quality model equations (Schultz, et al. 2000, 2002, 2004a,b). The RFM reduces simulation time from several days to several seconds and provides flexibility and speed that enables watershed managers to test assumptions and obtain useful insights into the response of regulatory performance metrics to policy choices. Because the RFM implicitly preserves NWPCAM’s original input data and the representation of hydrologic, chemical, and physical processes, it achieves integration of ecological, physical, and social performance measures.
Insights Gained Developing the RFM
Several insights can be obtained through application of the RFM that cannot be achieved using NWPCAM. Economic theory suggests that net benefits to society are maximized when the pollution controls are adjusted so that the marginal benefits to society equal the marginal costs of control. Policy analysts, however, frequently lack the functions needed to carry out such analyses and therefore compare incremental benefits with incremental costs for regulatory scenarios. This can lead to sub-optimal choices because benefits should be optimized in terms of the margin, not the increment. An advantage of the RFM is that it allows explicit identification of the marginal benefit function of pollution controls for inclusion as a component of an economic benefit function. Another important enhancement to the capabilities of NWPCAM that arises from the RFM and its closed-form solutions is the ability to implement prescriptive analyses that solve for the level of controls needed to achieve policy goals.
Coefficients of the RFM facilitate generalizations about pollution control strategies and the relative value of pollution controls at various sources. Simulation of the RFM is very rapid, providing the flexibility and speed needed to implement uncertainty analysis on physical or economic outputs and the ability to observe the sensitivity of water quality responses to regulatory controls under uncertain environmental conditions and modeling assumptions. In addition, it allows regulatory decisionmakers to investigate the assumptions of their cost-benefit analyses, a task that usually requires repetitive simulations. Specific insights are discussed below in the section on linkages between ecological, physical, and social components.
Ongoing Watershed Research
In the last phase of the research project, a new effort was initiated to study the possible contribution of improved information on future land use and climate change for improving water quality management decisions. This research, supported with leveraged funding from this project and EPA funding for the Consortium for Atlantic Regional Assessment is exploring water quality management decisions for the Ausable River within the New York State Adirondack Park. The Ausable River, famous for its fisheries and scenic beauty, is linked to the tourism and recreational industries of small towns such as Wilmington, New York. Water quality is threatened, however, by municipal waste and stormwater runoff associated with development in the area. A wet-weather water quality model is being developed for the waterway to assist with decision support in managing these waste sources and to help determine the extent to which water quality can be maintained as further facilities for tourists are developed. The importance of information on future land use and climate is being explored within the context of this decision-support study.
Linkages Among the Ecological, Physical, and Social Components of the Grant
Linkage among ecological, physical, and social components is demonstrated through two studies supported under this grant. The first is an application of a RFM that was fitted to NWPCAM to evaluate pollution control policies. The second is based on a study to develop water pollution trading systems, including the calculation of water pollution trading ratios.
Application of an RFM to Evaluate Regulatory Policy. Sensitivity and uncertainty analysis studies were conducted for both the water quality predictions of NWPCAM and the calculated economic benefits associated with the improvements in water quality from historic and possible future industrial, municipal, and nonpoint source treatment programs. Because of the complexity involved in implementing NWPCAM, however, a direct full-scale sensitivity and uncertainty analysis of the model for a range of treatment levels is not possible. To make such an analysis feasible, an RFM was developed to serve as a simplified representation of the full model, as described in Schultz, et al. (2004a and 2004b). Integration is achieved by incorporating the RFM function as a term in the calculation of performance metrics and economic benefit estimates. Figure 1 shows alternative realizations of a function for the performance metric, pB, the fraction of non-boatable stream miles changing to or from a boatable use level. These alternatives include a nominal case and five numbered alternatives showing results for different values of model input parameters. Two runs of the numerical model are required to observe a single point in this plane, so such insights are not readily available from the original model. Our project employs the integrative capabilities of the RFM to model responses to pollution controls and to characterize uncertainty in the benefit estimate.
Figure 1. Integration With Performance Metrics
RFMs were developed and applied for four states: Arizona, Iowa, Maryland, and Pennsylvania. Tables 1 and 2 depict key results from this evaluation. As shown in Table 1, further across-the-board reductions of 40 percent in loadings of conventional pollutants (TSS, BOD, TKN, and FC) at municipal treatment plants are unlikely to be cost-beneficial in any of the four states considered; however, Table 2 indicates the higher potential for net benefits from 85 percent reductions in combined sewer overflow (CSO) loads, especially in Maryland and Pennsylvania. In all cases, the uncertainties in the estimated benefits are indicated to be quite high, especially for the predicted benefits from CSO controls.
| Estimate | Arizona | Iowa | Maryland | Pennsylvania |
Benefit* | E[Benefit] | 8 | 33 | 137 | 526 |
s[Benefit] | 5 | 22 | 79 | 290 | |
90%-CI | 1 – 17 | 1 – 72 | 26 – 288 | 100 – 1063 | |
Cost** | 729 | 137 | 317 | 926 | |
* Benefits, estimated using the fitted response surface model, represent the present value of annual benefits over the 20-year CWANS planning horizon. E[Benefit] is the mean value, s [Benefit] is the standard deviation, and 90%-CI is the 90 percent confidence interval of the uncertainty distribution of the predicted benefits. | |||||
** Cost estimates from EPA’s 1996 CWANS (U.S. EPA, 1997). |
(Table 7.1 in Schultz, 2002)
| Estimate | Arizona | Iowa | Maryland | Pennsylvania |
Benefit* | E[Benefit] | 0 | 154 | 478 | 2,275 |
s [Benefit] | - | 202 | 506 | 2,163 | |
90%-CI | - | 1 – 541 | 2 – 2,973 | 147 – 6,849 | |
Cost** | 0 | 475 | 114 | 3,978 | |
* Benefits, estimated using the fitted response surface model, represent the present value of annual benefits over the 20-year CWANS planning horizon. E[Benefit] is the mean value, s [Benefit] is the standard deviation, and 90%-CI is the 90 percent confidence interval of the uncertainty distribution of the predicted benefits. | |||||
** Cost estimates from EPA’s 1996 CWANS (U.S. EPA, 1997). |
(Table 11 in Schultz, et al., 2004a)
Study and Development of Water Pollution Trading Systems. In addition to the benefit estimates derived from NWPCAM, the project has explored the use of the model to support the design of water quality emissions trading programs. The EPA Office of Water has released its “Final Water Quality Trading Policy” (EPA, 2003) noting that the Agency “. . . believes that market-based approaches such as water quality trading provide greater flexibility and have potential to achieve water quality and environmental benefits greater than would otherwise be achieved under more traditional regulatory approaches.” A key issue in establishing effective water quality trading programs is the determination of a trading ratio for discharges in different locations, of different types (e.g., point vs. nonpoint sources), and possibly of different pollutants as well. In previous work, our project researchers used a detailed watershed study of the Clear Creek watershed, Colorado, to investigate the determination of optimal point-nonpoint source trading ratios needed to meet copper and zinc standards at targeted monitoring stations in the watershed at a minimum cost (Schultz and Small, 2001). Although detailed studies such as this may be required to finalize within-watershed trading ratios, enabling widespread adoption of trading to allow more efficient attainment of environmental quality requires the identification of a simpler, more broadly applicable approach.
The basic approach developed in this research is based on the result that, with the optimal allocation of water pollution control resources, the ratios of the marginal costs of control at each source must be equal to the ratios of the unit damages caused by these sources. The ratio of unit damages thus serves as an appropriate starting point for determining the trading ratio for discharges from different sources (with additional, site-specific objectives, such as non-degradation and non-backsliding of current water quality, being used to constrain trades). A key output of our research has been the formulation of a modeling approach to determine unit damages and appropriate trading ratios for different water pollution sources (Farrow, et al., 2004). The method recognizes that for all pollutants, concentrations at the point of discharge are proportional to the discharge loading rate divided by the streamflow rate, but that impacts at points downstream depend on the subsequent mixing, dilution, and reaction processes that take place along the stream channel, and these effects can differ for different contaminants. Furthermore, damages are assumed to be equal to the product of the water quality impact and the number of affected households (which recognizes the contribution of local and national valuations of water quality for each stream segment). With this, the ratio of damages di from source i emissions of a given pollutant, to the damages dj from emissions of the same pollutant from another source, j, can be computed as (Farrow, et al., 2004):
(1)
where,
Ni is the number of river segments downstream of source i;
Nj is the number of river segments downstream of source j;
Hj is the number of households impacted by the river segments downstream of source j;
C0i is the pollutant concentration attributed to i at the point of discharge;
Cni is the pollutant concentration attributed to i at downstream river segment n;
C0j is the pollutant concentration attributed to j at the point of discharge;
Cnj is the pollutant concentration attributed to j at downstream river segment n;
Q0i is the average streamflow rate at the point of discharge for source i; and
Q0j is the average streamflow rate at the point of discharge for source j.
The use of equation 1 for determining trading ratios is illustrated by Farrow, et al. (2004) for CSO controls in the Upper Ohio River Basin (UORB), shown in Figure 2. The trading ratios are summarized in Table 3. For example, the ratio of household-weighted damage coefficients for Morgantown, West Virginia, relative to McKeesport, Pennsylvania, is 4.12 (cell 4,3). This ratio indicates Morgantown would have to purchase 4.12 units of emissions reduction at McKeesport to offset one unit of its own emissions. A unit load trade in the opposite direction is governed by the inverse of this ratio, 0.30. These differences are driven by the lower instream flow rates at Morgantown relative to McKeesport, differential downstream fate and transport, and different numbers of affected households (see equation 1). The highest ratio is 132.4 (cell 2,6) for Greensburg, Pennsylvania, obtaining offsets at Steubenville, Ohio. Every unit of emissions offset at Greensburg by emissions reductions at Steubenville would require 132.4 units at Steubenville.
These results and related work now are being considered as input by EPA in its development of an exploratory pollutant emissions trading program for the UORB, especially focused on nutrient control.
Figure 2. UORB Study Area. The eight largest combined sewer systems within the UORB lie within a 60-mile radius of the confluence of the Allegheny and Monongahela Rivers. The eight CSOs include: (A) Youngstown, (B) Steubenville, (C) Pittsburgh, (D) McKeesport, (E) Clairton, (F) Greensburg, (G) Uniontown, and (H) Morgantown. The inset shows the extent of the drainage area. (Reproduced from Farrow, et al., 2002.)
| Source j: Source of Pollution Offsets | ||||||||
Regulated Source i | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | |
1 | Clairton | 1.00 | 0.42 | 2.63 | 0.77 | 2.79 | 35.1 | 0.46 | 0.72 |
2 | Greensburg | 3.72 | 1.00 | 9.88 | 2.21 | 10.50 | 132.4 | 1.63 | 2.62 |
3 | McKeesport | 0.38 | 0.16 | 1.00 | 0.30 | 1.06 | 13.36 | 0.18 | 0.28 |
4 | Morgantown | 1.56 | 0.50 | 4.12 | 1.00 | 4.37 | 55.1 | 0.70 | 1.11 |
5 | Pittsburgh | 0.36 | 0.15 | 0.94 | 0.28 | 1.00 | 12.60 | 0.17 | 0.26 |
6 | Steubenville | 0.03 | 0.01 | 0.07 | 0.02 | 0.08 | 1.00 | 0.01 | 0.02 |
7 | Uniontown | 2.18 | 0.84 | 5.75 | 1.58 | 6.10 | 76.8 | 1.00 | 1.57 |
8 | Youngstown | 1.39 | 0.56 | 3.65 | 1.04 | 3.87 | 48.8 | 0.64 | 1.00 |
Evaluation of Broad Strategies Employed by the EPA for Water Quality Management. The management of water quality using approaches, such as the Clean Water Act Section 303(d) total maximum daily load process, requires an effective mix of modeling studies and monitoring data. In a paper supported by this study, Cooter (2004) reviews the capacity of current state assessment monitoring programs to support this effort, and demonstrates the critical role that broad-scale modeling efforts (such a NWPCAM) can play in complementing and extending current monitoring programs.
Journal Articles on this Report : 4 Displayed | Download in RIS Format
Other project views: | All 16 publications | 10 publications in selected types | All 4 journal articles |
---|
Type | Citation | ||
---|---|---|---|
|
Cooter WS. Clean Water Act assessment processes in relation to changing U.S. Environmental Protection Agency management strategies. Environmental Science & Technology 2004;38(20):5265-5273. |
R828021 (Final) |
Exit Exit |
|
Farrow RS, Schultz MT, Celikkol P, Van Houtven G. Pollution trading in water quality limited areas: use of benefits assessment and efficient trading ratios. Land Economics 2005;81(2):191-205. |
R828021 (Final) |
not available |
|
Schultz MT, Small MJ, Farrow RS, Fischbeck PS. State water pollution control policy insights from a reduced-form model. Journal Of Water Resources Planning And Management-Asce. 2004;130(2):150-159. |
R828021 (Final) |
not available |
|
Schultz MT, Small MJ, Fischbeck PS, Farrow RS. Evaluating response surface designs for uncertainty analysis and prescriptive applications of a large-scale water quality model. Environmental Modeling and Assessment. |
R828021 (Final) |
not available |
Supplemental Keywords:
science models, spatial aggregation, water quality, water quality model, watershed, environmental managers, nutrients, toxics, modeling tool, watershed managers, regulatory agencies, performance measures,, RFA, Scientific Discipline, Water, Water & Watershed, Wet Weather Flows, Environmental Monitoring, Ecological Risk Assessment, Ecology and Ecosystems, Social Science, Watersheds, social impact assessment, watershed, social performance measurement, statistical approaches, multiple monitoring sites, decision making, cost benefit, decision model, integrated watershed model, water quality, economic benefit, National level, statistics, aquatic ecosystems, wet weather modelingProgress and Final Reports:
Original AbstractThe perspectives, information and conclusions conveyed in research project abstracts, progress reports, final reports, journal abstracts and journal publications convey the viewpoints of the principal investigator and may not represent the views and policies of ORD and EPA. Conclusions drawn by the principal investigators have not been reviewed by the Agency.