EPA's Report on the Environment: External Review Draft
Frequent QuestionsWhat is the ROE? ROE Development
- How are ROE questions developed?
- How are ROE indicators selected?
- Has the ROE been scientifically reviewed?
- Where do ROE indicator data come from?
- What technical documentation is provided for ROE indicators? How can I access this documentation?
- What statistical information is presented for ROE indicators?
- What are confidence intervals? Why are they provided for ROE indicators (when available)?
- What trend information does the ROE present?
- What other information does the ROE present relevant to indicator uncertainty or limitations?
- Why was sustainability added as a new ROE theme in 2014?
- What was EPA's strategy for expanding the focus of the ROE to include sustainability?
- How were the four new sustainability indicators selected?
- Why does the sustainability theme have just one ROE question?
- How do the sustainability indicators differ from other ROE indicators, and how should one interpret them?
- How do the ROE indicators relate to other scientific indicator reports by EPA?
- How does the ROE relate to EPA's strategic plan for FY 2014-2018?
What is the ROE?
Is the ROE a report card on the environment?
The ROE is not a “report card” on the state of the environment. Rather, it uses indicators to address specific questions about trends in the environment and human exposure and health.
Is the ROE an assessment?
The ROE is not an assessment. Many other environmental reports, particularly those that focus on particular issues or locations, conduct integrated assessments by gathering and weighing the strengths and weaknesses of all the relevant information available. The ROE intentionally focuses only on indicators, thus by definition is not a comprehensive assessment because it does not include other data. The trends provided in the ROE may be used by others as part of an assessment. Questions may be added or modified over time to reflect changes in EPA priorities.
How are ROE questions developed?
ROE questions are those that EPA believes should be answered with confidence for the Agency to be adequately informed about important environmental trends, but they are not necessarily questions that EPA can fully answer at present.
To develop ROE questions, EPA convenes scientists and other specialists from across the Agency to identify questions of importance to EPA's mission. ROE questions are posed without regard to whether indicators are available to answer them. Questions may be added or modified over time to reflect changes in EPA priorities.
How are ROE indicators selected?
Once an ROE question is developed, EPA determines what indicators are suitable to help answer the question, and what data are available to develop these indicators. Some ROE indicators help to answer more than one question.
To identify potential indicators, EPA solicits suggestions from EPA specialists, other federal agencies, state agencies, non-governmental organizations, and academic experts. EPA then screens the candidate indicators against the indicator definition and criteria, and develops the most needed indicators when suitable data are available. Draft indicators are peer-reviewed by scientists outside the Agency before being added to the ROE.
Though the ROE presents many indicators, additional indicators are still needed to fully answer many of the ROE questions. For one question, no ROE indicators are available. EPA works with federal, state, and nongovernmental organizations to identify potential new indicators and data on an ongoing basis.
Has the ROE been scientifically reviewed?
Yes. All ROE indicators are externally peer-reviewed by experts in relevant scientific fields to ensure that they meet the ROE indicator definition and criteria, are useful for answering the ROE questions, and are properly documented and scientifically sound. Only indicators that pass this review are included in the ROE. In addition, the entire report is reviewed periodically by EPA's Science Advisory Board (SAB). See Peer Review and Science Advisory Board (SAB) Reports for more information about these reviews and their findings.
Where do the ROE indicator data come from?
The underlying data for the ROE indicators were collected and compiled by a variety of federal and state agencies, including EPA, USGS, NOAA, and NASA, and non-governmental organizations, such as NatureServe . Each indicator includes a “Data Sources” section describing the source(s) of the indicator data.
Technical Documentation and Statistical Information
What technical documentation is provided for ROE indicators? How can I access this documentation?
For each indicator, you can click on the “View Technical Documentation” link toward the bottom of the indicator page to view a technical documentation form describing:
- Data sources and availability.
- Methodology underlying the indicator, including data collection, indicator derivation, and quality assurance/quality control procedures.
- Analysis, including reference points, comparability over space and time, sources of uncertainty and variability, and information on statistical/trend analysis.
- Data limitations.
What statistical information is presented for ROE indicators?
The ROE displays confidence intervals (in the form of error bars or bands) and trend lines as optional overlays to indicator graphics when such data are readily available and straightforward to add to the graph. Currently, the ROE displays confidence intervals for ten indicators and trend lines for one indicator. The presence of the statistical information button to the right of a graph signals the availability of confidence intervals or trend lines that can be added to the display. Click on this button to view the available statistical information.
Additional details about uncertainty, variability, and statistical analysis (for example, analysis of trends over time) can be found in the technical documentation for each indicator. A footnote beneath every ROE exhibit explains what is known about any trends in the data over time, if applicable.
What are confidence intervals? Why are they provided for ROE indicators (when available)?
ROE indicators are based on measurements of environmental and human health data. The possibility of errors in sampling, measurement, and/or reporting processes, as well as random variation associated with the underlying activity being measured, can introduce uncertainties into environmental measurement data. In other words, the actual value in the environment may be more or less than the measured value.
When sufficient information is available, statisticians can quantify the degree of uncertainty associated with measured values as a confidence interval. The key elements of a confidence interval are the confidence limits, or boundary values of the interval, and the associated probability that the actual value is contained in that interval. For example, when one refers to a 95 percent confidence interval, it means there is a 95 percent probability that the actual value falls within the range of the confidence interval. For ROE indicators, confidence intervals are displayed visually in the form of error bars on a bar graph or a shaded band on a line graph, as shown in the examples below.
- Error bars. Error bars show the upper and lower bounds for each data point expressed as a confidence bound or standard deviation. In the Blood Lead Concentrations exhibit shown below, the error bars show the range associated with the 95 percent confidence interval. This means there is a 95 percent probability that the true (i.e., error-free) measurement value lies somewhere within the range bounded by the error bars.
- Confidence bands. Other indicators, such as Sea Surface Temperature shown below, use a confidence band to display the confidence interval. A confidence band represents the uncertainty associated with a curve or function estimate. Like the error bars, the confidence band shows the range associated with the 95 percent confidence interval, meaning there is a 95 percent probability that the true value for each year lies somewhere within the range of the band.
What trend information does the ROE present?
Data presented over time, as in many of the ROE indicators, are referred to as a time series. A time series may look like it is increasing or decreasing over time, but statistical techniques are needed to calculate the slope of this trend and determine whether it truly represents an increase or decrease. These techniques are called trend analysis. A footnote beneath every ROE exhibit explains what is known about any trends in the data over time, if applicable. See the footnote beneath the Sea Surface Temperature exhibit above for an example.
Trends are often calculated using ordinary least squares regression, which involves finding the straight line that best fits the shape of the time series. This line may slope upward, slope downward, or remain nearly flat. The slope of this line represents the slope of the trend over time. It can also be thought of as the average rate of change (for example, temperature may increase at an average rate of 0.013°F per year).
The next step in trend analysis is to determine whether the trend is statistically significant. Just as a measured value may differ from the actual value because of various forms of error, the calculated slope may differ from the actual trend because of factors such as variability in the data or a limited number of observations. Statistical significance refers to the likelihood that the actual slope is in the same direction as the calculated slope. Significance is expressed in terms of a percentage. For example, if an increasing trend is said to be “significant to a 95 percent level,” that means there is at least a 95 percent chance that the actual trend is an increase. In general, the steeper the slope, the more data points available, and the less variability they show from year to year, the more likely it is that a trend will be statistically significant. In contrast, a trend is more likely to be insignificant if the calculated trend is close to zero, if there is a lot of year-to-year variation in the data, or if there are so few data points that it is difficult to feel confident about the trend.
The ROE considers trends to be statistically significant if they reach a level of confidence that is generally viewed as appropriate for the topic at hand. In most cases, the threshold is 95 percent confidence. Where feasible, trend lines may be displayed in ROE exhibits for trends that are statistically significant.
Currently, the ROE displays trend lines for one indicator: Temperature and Precipitation. Click on the statistical information button to the right of each exhibit to view these statistically significant trends. To see the slope of the trend, view the legend or the tooltip that appears when hovering over the line. The technical documentation describes how each trend was calculated and provides the p-value, which indicates the level of significance. In a few cases, the technical documentation may explain that a trend was calculated but determined to be insignificant, in which case no trend line is shown on the graph.
Trend analysis in the ROE has several key limitations:
- Many ROE indicators do not yet have enough data points over time to support trend analysis.
- Trend analysis cannot explain underlying processes or provide any information on why a particular trend is observed.
- At present, most of the ROE trend analysis is restricted to the most common type of analysis: ordinary least squares. Other more sophisticated forms of analysis are available, including techniques that consider serial correlation. Serial correlation analysis of trends in ROE indicators may be presented in the future.
What other information does the ROE present relevant to indicator uncertainty or limitations?
For all ROE indicators:
- The indicator text describes the indicator's limitations.
- The indicator's technical documentation provides an overview of sources of uncertainty and variability, as well as information on statistical/trend analysis (when available).
Why was sustainability added as a new ROE theme in 2014?
In 2010, former EPA Administrator Lisa P. Jackson commissioned the National Academy of Sciences to provide recommendations on an operational framework for sustainability for EPA. That study led to the 2011 NAS/NRC report, Sustainability and the U.S. EPA (also known as the Green Book). This report strongly urges EPA to adopt a sustainability framework to advance methods and approaches that will help EPA and others make better and more informed environmental decisions. EPA is currently developing its response to the Green Book recommendations.
As part of EPA's shifting focus to sustainability, EPA's Office of Research and Development (ORD) adopted a sustainability paradigm to guide its research programs, and Paul Anastas, former ORD Assistant Administrator, tasked the ROE program with incorporating sustainability indicators to reflect the Agency's expanding interest in this important area.
What was EPA's strategy for expanding the focus of the ROE to include sustainability?
The Agency used a three-pronged approach that consisted of: (1) adopting a systems-based sustainability framework as the overarching conceptual model for the ROE, (2) adding a new sustainability theme that focused on the question of consumption of natural resources, and (3) developed indicators to help address that question.
How were the four new sustainability indicators selected?
An ORD Sustainability Indicators Workgroup was tasked with exploring the universe of existing sustainability indicators and recommending which indicators were most appropriate for the ROE. The indicators were designed to be broadly informative about the intensity of use of specific resources relevant to EPA's mission. Based on these recommendations, the Agency developed four indicators, all of which were independently peer-reviewed before being added to the ROE. The new indicators are intended to help the Agency explore three important aspects of sustainability—intensity of energy use, water use, and materials use—and understand how Agency decisions and actions can best advance sustainability.
Why does the sustainability theme have just one ROE question?
EPA is taking a phased approach to developing this theme and incorporating new indicators. Over time, EPA will expand the sustainability theme by including additional questions and indicators to help address the questions. EPA plans to work collaboratively with a broad range of stakeholders to improve and foster the use of sustainability indicators to aid all parties in more efficient and effective protection of human health and the environment.
How do the sustainability indicators differ from other ROE indicators, and how should one interpret them?
All ROE indicators present trends in environmental stressors or conditions, and a number of them describe how economic or social factors can influence those trends, but only the four sustainability indicators quantify that influence.
The ROE sustainability indicators present economic and population data to provide a more complete picture of how underlying drivers are influencing observable outcomes. For example, in the Freshwater Withdrawals indicator, Exhibits 1 and 2 present trends over time in total nationwide withdrawals of fresh water, while Exhibit 3 shows patterns of freshwater withdrawals per dollar of real gross domestic product and per capita. In other words, Exhibit 3 of this indicator uses intensity metrics to examine the role of economic growth and population growth as drivers of natural resource consumption (fresh water in this case).
The intensity metrics presented in the sustainability indicators cannot fully answer the question of whether U.S. resource use is truly "sustainable" by every definition of the word—a fact made clear in the "Limitations" section of each indicator. However, the ROE intensity metrics do provide useful information on the extent to which the United States has decoupled resource use from economic and population growth.
Relationship to Other EPA Reports/Plans
How do the ROE indicators relate to other scientific indicator reports by EPA?
In addition to the ROE, EPA publishes other sets of indicators to report on specific issues of concern, such as climate change, environmental conditions in the U.S./Mexico border region (PDF) (106 pp, 2.3M), and children's health and the environment. While these other EPA products may share some indicators in common with the ROE, they also provide additional indicators beyond what appears in the ROE. Many of these additional indicators are of the same high quality as the indicators in the ROE, and they have not been included in the ROE simply because the ROE aims for a wide breadth of topics rather than deep coverage of a single topic. These additional indicator compilations are frequently mentioned in the “Related Links” and “For More Information” links at the bottom of ROE web pages. EPA encourages readers to explore these additional indicators to learn more about specific topics of interest.
How does the ROE relate to EPA's strategic plan for FY 2014-2018?
EPA's Strategic Plan provides the foundation for the Agency's performance management system—planning, budgeting, performance measurement, and accountability. Published every four years, the Plan describes the Agency's long-term goals, objectives, and strategic measures, and it serves as a management tool for EPA's senior leadership to establish priorities, manage programs, and measure and report performance in protecting human health and the environment.
EPA's Report on the Environment, in contrast, is a science–based tool that uses the best available indicators to monitor trends in the condition of the nation's air, water, and land, and associated trends in human exposure and health, the condition of ecological systems, and sustainability.
Although these two publications serve different purposes, they are related in several ways. The environmental trends shown in the ROE can help to inform strategic planning, priority setting, and decision-making across EPA. Because both the Plan and the ROE are rooted in EPA's mission to protect human health and the environment, the five Strategic Plan goals and the six ROE themes are highly related:
|Strategic Plan Goals||Corresponding ROE Themes|
|Goal 1: Addressing Climate Change and Improving Air Quality||Air, Human Exposure and Health, Ecological Condition|
|Goal 2: Protecting America’s Waters||Water, Human Exposure and Health, Ecological Condition|
|Goal 3: Cleaning Up Communities and Advancing Sustainable Development||Land, Human Exposure and Health, Ecological Condition, Sustainability|
|Goal 4: Ensuring the Safety of Chemicals and Preventing Pollution||Land, Human Exposure and Health, Ecological Condition|
|Goal 5: Protecting Human Health and the Environment by Enforcing Laws and Assuring Compliance||This Agency goal involves administrative measures outside the ROE scope.|
For each goal, EPA's Strategic Plan sets forth specific objectives and strategic measures describing the human health and environmental results EPA is working to achieve. Several of the objectives and strategic measures in EPA's Strategic Plan draw upon or are otherwise closely related to ROE indicators. For example, the Strategic Plan and the ROE both report on the following environmental variables:
- Emissions and concentrations of air pollutants, including greenhouse gases, ozone, particulate matter, air toxics, and sulfur dioxide.
- Acidity in lakes and streams.
- Radon exposure.
- Populations served by community water systems with no reported violations of health-based standards.
- The condition of the nation’s rivers and streams, lakes, and coastal water.
- Wetland extent.
- Hypoxia in the Gulf of Mexico and Long Island Sound.
- Waste generation and management.
- Contaminated groundwater migration and human exposure at cleanup sites.
- Pesticide exposure.
- Blood concentrations of toxic chemicals, including mercury and lead.
With its new Sustainability theme and indicators, the ROE is also aligned with one of the Agency's new Cross-Cutting Fundamental Strategies, “Working Toward a Sustainable Future.”