Grantee Research Project Results
Final Report: Estimation of Spatially Explicit Water Quality Benefits throughout River Systems: Development of Next Generation Stated Preference Methods Using National Probability Samples and Online Labor Pools
EPA Grant Number: R836167Title: Estimation of Spatially Explicit Water Quality Benefits throughout River Systems: Development of Next Generation Stated Preference Methods Using National Probability Samples and Online Labor Pools
Investigators: Johnston, Robert J , Wollheim, Wil , Moeltner, Klaus
Institution: Clark University , University of New Hampshire , Virginia Tech
EPA Project Officer: Packard, Benjamin H
Project Period: April 1, 2016 through March 31, 2019 (Extended to March 31, 2022)
Project Amount: $799,919
RFA: Water Quality Benefits (2015) RFA Text | Recipients Lists
Research Category: Water
Objective:
This project developed novel approaches for stated preference valuation designed to address the challenges of total willingness to pay (WTP) estimation for complex, spatially explicit aquatic ecosystem change with heterogeneous benefits for different user and nonuser groups. The integrated approach restructures the way that WTP is elicited and estimated in survey-based valuation, hybridizing methods from contingent valuation, choice experiments, revealed preference modeling, GIS map tracking, and Bayesian econometrics. It was developed to estimate WTP for water quality improvements throughout river networks, and applied to a case study of a large river system spanning six New England states. The developed models and methods were designed to be transformative across multiple dimensions, including: (a) novel ways to elicit stated preferences and design surveys, with scenarios coupled directly to ecological models and interactive GIS maps, (b) flexible value elicitation and modeling that generates benefit functions linked to spatially explicit effects, (c) an ability to evaluate the relevance of multiple ecological indicators to different groups, and (d) novel modeling of the data using Bayesian econometrics. Biogeochemical forecasts for valuation scenarios were projected using FrAMES (Framework for Aquatic Modeling of the Earth System), a process-based spatially explicit water quality model. The developed methods provide a means to estimate economic values for a variety of surface waters and types of quality change. Particular attention was given to the capacity of this new approach to identify whether and how households’ WTP for different types of water quality changes varies according to the spatial dimensions of those changes, including forms of spatial welfare heterogeneity that are unidentifiable using traditional choice experiment methods. We also compared results derived from push-to-web population sampling to an alternative approach using online labor pools and panels.
Summary/Accomplishments (Outputs/Outcomes):
The project developed a novel approach to stated-preference valuation, based on direct integration of a water quality model and a map-based, interactive choice experiment. We used the approach to estimate households’ WTP for water quality improvements over approximately 95,800 miles of rivers and streams in Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, and Vermont (71,992 sq. mi. of watershed area). Using these methods, we estimated WTP for realistic, predicted scenarios of water quality change from a set of possible policy actions throughout this river system. Among the primary empirical goals of the study was to provide insight on the extent to which WTP is determined by the spatial distribution of different types of water quality changes throughout the river network, focusing on household-specific, spatial determinants of value that are unmeasurable using standard approaches. Innovative elements of the study included a novel, interactive approach to GIS-map interaction within the valuation questionnaire. Voting scenarios used to elicit WTP included interactive GIS maps that illustrated three water quality measures at various zoom levels across the study domain. The online questionnaire captured data on how respondents maneuvered through these maps prior to answering the value-eliciting questions. We hypothesized that allowing people to interact with GIS maps in a valuation survey in this manner might provide information that improves the accuracy of WTP prediction, by identifying individualized areas where water quality improvements have high value to each respondent.
Among the primary advances pioneered by the project was elicitation of WTP using ecologically realistic scenarios of water quality change that dispensed with the oversimplified, ecologically artificial “matrix” format that is ubiquitous in contemporary choice experiments. The methods developed by this project purposefully avoided key simplifications that are common in choice experiment scenarios (e.g., limiting scenarios to “average changes” over large areas). The survey elicited values for a set of predicted water quality changes throughout a regionwide river system—such that changes potentially occurred everywhere throughout the domain and were not focused on iconic or recreational areas. This less restricted form of scenario allowed respondents to explore heterogeneous water quality changes using interactive maps, and thereby ground their choices in changes and areas that matter to them—rather than having these predefined by the researchers. This structure, paired with map-interaction architecture, allowed us to characterize spatial dimensions of WTP for realistic scenarios of water quality in ways not possible otherwise.
The survey elicited choices for WTP estimation using choice experiment (policy) scenarios that each reflected a possible set of water quality improvements. Each respondent received a questionnaire presenting one of these scenarios. The scenarios presented in the survey were designed around hypothetical but realistic water quality changes that could occur throughout the river and stream system as of 2025, as a function of illustrative policy measures that could be applied to improve water quality. These scenarios were developed for purposes of eliciting WTP linked to a range of possible future scenarios of water quality change. Scenarios depicted possible changes in the three water quality measures: safety for human use (WS), support for aquatic life (AL), and multi-metric overall water pollution (WQ). The water safety (WS) indicator for human use was developed based on fecal coliform guidelines. The indicator representing support for aquatic life (AL) was developed based on effects of chloride concentrations on aquatic organism survival. The indicator for total water pollution (WQ) was developed as a combined metric using all modeled solute concentrations. Each measure was normalized to a 0 (worst) to 100 (best) scale, based on reference conditions for the domain and biophysical thresholds for each indicator. We also binned each indicator into seven color-coded and labeled intervals. For each scenario, the survey communicated each quality measure in three different ways: (a) a normalized, spatial mean value (0 - 100) over the river system, (b) a bar chart showing the proportion of total river and stream miles within seven binned quality intervals, and (c) high-resolution GIS maps showing water quality predictions over the river system.
The primary online survey was implemented using an address-based, push-to-web sample, drawn randomly from households in the six sampled states. Each choice experiment voting question paired a possible environmental policy scenario with a hypothetically binding household cost required to implement the scenario, compared to a “business as usual” (BAU) status-quo with no change in household cost. Each scenario included a spatially explicit prediction of water quality changes over the study domain, produced using FrAMES (Framework for Aquatic Modeling of the Earth System), a process-based water quality model. These measures were communicated in multiple ways, including interactive GIS maps that enabled each measure to be viewed at various zoom levels across the river system. The survey architecture captured data on how respondents maneuvered through each interactive map prior to answering choice experiment questions. Map-interaction data was used to infer where and at what scale water quality might be relevant to each respondent and provided evidence of increased attention to particular areas.
The subsequent value-elicitation question was a single, hypothetically binding, binary vote between the BAU and the presented policy scenario, designed for incentive compatibility. Each voting question compared predictions for the three water quality measures under the BAU status quo and one of the possible policy scenarios, paired with a hypothetically binding, annual cost per household (in unavoidable taxes and fees) that would be required to obtain the alternative policy scenario. Possible cost levels for the policy scenario were $30, $60, $120, $240, $480, $720, $960 and $1200. To produce each voting question, one of the 41 possible environmental scenarios was paired with one of the eight possible cost levels. The additional cost of the BAU was always $0. The pairing algorithm in the online survey was designed to produce a balanced number of survey responses across all possible scenario-cost pairs.
Survey materials were developed over a four-year period from September 2016 to April 2021. All materials, including scenarios and water quality measures, were developed and pretested with input from 7 focus groups and 20 cognitive interviews to ensure comprehension by non-expert respondents, with additional input from environmental scientists and experts in stated preference methods. Focus groups included a total of 54 non-expert adult subjects (7-8 per group), identified via random recruitment by external firms and paid for participation. Four additional online pilot tests were used to evaluate and adjust the survey prior to implementation. Respondents for the first two pilot tests (N = 200, 136) were recruited through the Amazon Mechanical Turk (MTurk) online labor pool during September – October 2020. Respondents for the second two pilot tests (N = 136, 60) were recruited by an external vendor (Qualtrics) during November 2020. Pilot test data was analyzed to evaluate response patterns, the sensitivity of votes to the environmental conditions and cost, symptoms of potential protest responses, responses to follow-up questions (after the primary voting question), indicators of data quality, the extent to which subjects engaged with maps, and other validity and response diagnostics. Prior approvals for all human subjects research were obtained from the IRBs of Clark University and Virginia Tech, and the EPA Human Subjects Research Review Official.
The primary push-to-web survey was implemented during May to June 2021. The survey was implemented using an address-based push-to-web sample, with invitations mailed to 7,167 randomly selected names/addresses from each sampled state (Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, and Vermont). The mailing sequence included an initial letter introducing the survey, requesting a response and providing the survey URL (website link) along with a unique identification number and password for each respondent. This initial letter was followed, at weekly intervals, by two reminder letters. Of 42,979 deliverable invitations, 2,203 total responses were received (5.13% response rate). These included some responses that did not provide sufficient data for inclusion in choice modeling. Of all responses, 1,698 answered the main choice question and had an identifiable home location in or close to the study area, and were hence eligible for inclusion in one or more of the estimated choice models. Of these, 1,292 respondents interacted with at least one map (76%), and 1,239 interacted with at least one map and lived within 10 miles of the study domain. These different subsets of respondents were used to support multiple alternative random-utility models used to produce our primary results and evaluate robustness.
For comparison, an alternative preliminary full sample was also drawn using an online, opt-in panel maintained by Qualtrics. This implementation was done using a subset of only 7 of the 41 experimental design profiles in the choice experiment. The online panel version of the (otherwise identical) survey was implemented during November 2020 – February 2021. Respondents were drawn from residents of the six New England states, with quotas for age, gender, income, and education to approximate regional Census distributions. Multiple quality screens were used to ensure response validity. After screening, the online panel sample yielded N = 3,374 observations. Comparison of results from the push-to-web sample and online panel sample suggested that the latter group of respondents (from the online panel) was less engaged with the survey (e.g., less likely engage with interactive survey maps) and spent much less time completing the survey (median response time of 13 minutes compared to 22 minutes for the push-to-web sample). Symptoms of low-quality and inattentive responses were also high for the online panel responses, as reported by Johnston et al. (2021). For this reason, the main analysis and results from the project rely solely on data from the primary, high-quality, push-to-web sample. The superior quality of the push-to-web sample was anticipated and suggests that samples of this type are more likely to produce high-quality data for choice modeling. The remainder of this final report focuses solely on results from the primary, high-quality, push-to-web sample.
Summary of Map-Interaction and Econometric Choice-Model Results
The analysis of map-interaction data was organized around the zoom (or magnification) level with which each respondent viewed each map extent (or frame). Many views in our dataset represent the “default” viewing extent at which maps load initially (zoom-level 6), or a re-centered view at a scale too generalized to convey new information (beyond information in the static maps included on the main survey screens viewed by all respondents). To orient our analysis around informative scales, we therefore disaggregated analysis of longest-looked frames according to standardized zoom-levels (6-10, 11-13, 14-17). Zoom levels are discrete, preset scales at which a map is pre-rendered on the screen, used by most modern interactive mapping platforms—analogous to the magnification at which each map was viewed. Levels 6-10 represent geographies between the country and city level (e.g., states, metropolitan areas), 11-13 represent communities, neighborhoods and roads, and 14-17 display smaller streets and structures. As anticipated, analysis of larger-scale views (zoom 6-10) displayed minimal spatial variation over the region; these views often covered the entire study area and thus did not demonstrate respondents’ intentions to view particular areas. Hence, we oriented the analysis around views at zoom-levels 11-13 and 14-17 (areas that respondents “zoomed in” to see at higher resolution).
Results for zoom-levels 11-13 show clear patterns of intra-regional spatial variation. Although map-interaction data reveals higher respondent interest in areas covering population centers, its pattern of variation is not fully explained by the distribution of respondent addresses, as would be the case if respondents only zoomed in to view their own homes or communities. Data for all longest-looked map frames verifies that only 56.8% of views at levels 11-13 included the respondent’s home within the viewed area. Similar patterns emerge for levels 14-17. Views at these levels are only captured for respondents who viewed highly specific areas of a map. Viewing frequency at this level also increases for general areas near respondents’ homes, but the relative density of map views and home addresses is not identical. Only 35.2% of frames viewed at levels 14-17 included the respondent’s home in the viewed area.
Using the choice experiment response data combined with the map-interaction data, discrete-choice, random-utility models were estimated in “WTP-space” using Bayesian model search, with parameters interpreted as dollar-denominated WTP estimates. The models predicted each respondent’s vote and corresponding WTP measures as a function of explanatory variables derived from spatially explicit water quality measures in the choice scenario, map interactions, and each respondent’s home location. Among other key findings, results show that WTP is influenced by regionwide changes in water quality measures and by the spatial distribution of these changes. With regard to the latter, WTP is influenced by the extent to which changes occur within (a) 10- and 25-miles of each respondent’s home and (b) the geographic area given the longest attention by each respondent during their map interactions, irrespective of home location. These effects vary over different quality measures and are most pronounced for improvements to areas at low baseline quality. Supplemental models were estimated to evaluate robustness and test for WTP heterogeneity associated with respondents who interacted with maps (versus those who did not). All results suggest the validity and reliability of the project’s primary results.
Conclusions:
Project findings provide robust evidence that the developed methods provide insights into water quality values that are unavailable though other types of stated preference valuation techniques. Combined evidence from survey development procedures (e.g., focus groups, cognitive interviews, pilot tests), data validity screens (e.g., evaluations of construct validity, validity-check questions, protest-response follow-up questions), geospatial map-interaction analyses, and econometric analyses supports the technical effectiveness and feasibility of the approach. Although illustrated for a single New England case study, these methods can be readily adapted to other contexts for which information on water quality benefits is required. Project results provide multiple new insights into how New England households value water quality change, including first evidence that respondents’ map interactions convey systematic information that is related to their choices and WTP estimates. Estimated values are influenced by water quality changes close to each respondent’s home, as anticipated, but also in locations identifiable via each respondent’s map interactions. These spatial effects are pertinent solely for improvements to rivers at low current quality, indicating that spatial WTP heterogeneity depends on whether improvements occur in high- or low-quality waters.
The project’s findings have direct implications for how people value water quality and how values are estimated to support environmental programs and policies. Households’ aggregated WTP can represent much of the total economic benefit of water quality improvements to society. WTP typically depends on what type of improvements occur and where. However, the ways in which spatial dimensions of water quality influence WTP are often modeled primarily with respect to where people live or iconic locations. This study shows that allowing people to interact with maps of water quality change during a choice experiment survey can reveal individualized areas wherein some types of water quality improvements have particularly high value, beyond effects related to home locations. Improvements in these areas are associated with significant increases in households’ WTP. Although many approaches are available to model spatial dimensions of WTP, stated preference data is almost always generated using survey architectures that provide little or no opportunity for respondents to engage with maps or explore conditions in areas that matter to them. We find that this engagement provides information that explains how and where people value water quality. The methods developed by this project are likely to be particularly relevant for obtaining valid and reliable information on water quality values when these values are influenced by geospatial dimensions in ways that might not be anticipated by researchers.
Journal Articles on this Report : 1 Displayed | Download in RIS Format
Other project views: | All 6 publications | 1 publications in selected types | All 1 journal articles |
---|
Type | Citation | ||
---|---|---|---|
|
Johnston R, Moeltner K, Peery S, Ndebele T, Yao Z, Crema S, Wollheim W, Besedin E. Spatial dimensions of water quality value in New England river networks. PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA 2023;120(18):e2120255119 |
R836167 (Final) |
Exit Exit |
Supplemental Keywords:
Media: water, watersheds; Ecosystem Protection: ecosystem, indicators, aquatic, habitat; Public Policy: decision making, cost-benefit, non-market valuation, contingent valuation, survey, preferences, public good, Bayesian, willingness-to-pay; Disciplines: social science, economics, ecology, hydrology; Methods/Techniques: modeling, analytical, surveys; Geographic Areas: Northeast, EPA Region 1; Other: ecosystem service, choice modeling, choice experiment, nonuse value, welfare analysis, water quality.Progress and Final Reports:
Original AbstractThe perspectives, information and conclusions conveyed in research project abstracts, progress reports, final reports, journal abstracts and journal publications convey the viewpoints of the principal investigator and may not represent the views and policies of ORD and EPA. Conclusions drawn by the principal investigators have not been reviewed by the Agency.
Project Research Results
- 2020 Progress Report
- 2019 Progress Report
- 2018 Progress Report
- 2017 Progress Report
- 2016 Progress Report
- Original Abstract
1 journal articles for this project