Grantee Research Project Results
Final Report: Optimal Experimental Design for Nonmarket Choice Experiments
EPA Grant Number: R827987Title: Optimal Experimental Design for Nonmarket Choice Experiments
Investigators: Kanninen, Barbara J.
Institution: University of Minnesota
EPA Project Officer: Chung, Serena
Project Period: January 3, 2000 through January 2, 2003
Project Amount: $61,014
RFA: Decision-Making and Valuation for Environmental Policy (1999) RFA Text | Recipients Lists
Research Category: Environmental Justice
Objective:
The objectives of this research project were to: (1) derive optimal designs for (hypothetical) choice experiments to get the best possible estimate for willingness to pay when substitution effects are present; (2) make the current literature and research of the Principal Investigator (PI) more relevant to environmental aluation in practice by using optimal design criteria that are directly linked to the goals of improving estimation of willingness to pay or marginal rates of substitution; (3) compare the statistical gains or losses associated with multinomial versus binary choice experiments and with the inclusion or omission of substitution effects; and (4) generate rules of thumb that can be easily communicated to experimental choice researchers to improve their survey designs in practice.
Summary/Accomplishments (Outputs/Outcomes):
Environmental economists often use choice experiments to assess people's willingness to pay for different environmental attributes. In choice experiments, people are shown a set of hypothetical goods and are asked to choose their most preferred option. The goods vary by attribute levels; for example, the numbers of particular wildlife species viewed, or measures of air or water quality.
Experimental design of choice experiments refers to the way attributes are assigned to choice sets. Before conducting a choice experiment, the researcher must assign levels for the different attributes and allocate the different levels across alternatives. The experimental design, combined with people's choice responses, make up the dataset that will ultimately be used to estimate values.
The optimal design literature has shown that choice set designs can greatly affect the amount of statistical information a dataset provides. Until the PI's work, however, the literature focused only on improving designs by re-working the attribute level allocations across choice sets. The PI's work took optimal design a significant step forward by looking at how the choice of attribute levels affects information.
There are many ways to measure statistical information. Optimal design researchers, therefore, must specify a certain design criterion to evaluate the quality of a design. The design criterion specifies precisely which statistical aspect of a set of estimators is most important to the researcher. One of the most popular design criteria is called D-optimality. This criterion looks for a dataset that jointly provides the most statistical information about the full set of model parameters.
In earlier research, the PI derived D-optimal designs for choice experiments. Though D-optimality is a useful, general criterion, the PI believed that for environmental valuation purposes, an alternative design criterion might be more useful. The first phase of the current project focused on a design criterion called C-optimality. This criterion focuses on getting the best estimate possible for willingness to pay.
The PI found that the C-optimal design solution is to generate choice sets that make respondents perfectly indifferent between alternatives. In other words, attribute levels should be set so there is zero utility difference between each alternative in a choice set.
Unfortunately, the PI found that in practice this solution results in multicollinearity: attribute levels in each alternative would be exact linear combinations of the other attributes. This multicollinearity occurs because there are more variables in the problem than are necessary for estimating willingness to pay. So under C-optimality, the model falls apart and is inestimable.
Based on this result, the PI concluded that the standard approach to estimating willingness to pay as a ratio of estimated parameters is inherently inefficient. From a purely statistical perspective, the best approach to estimating willingness to pay is to drop extraneous variables and estimate willingness to pay directly-in other words, just ask people their willingness to pay.
Note that this is the statistically best approach, but not necessarily the best for obtaining reliable responses. In fact, the research community highly favors the indirect approach to obtaining willingness to pay information. Because of this, the PI has concluded that the D-optimality criterion, rather than C-optimality, is the most appropriate basis for designing choice experiments.
An extensive set of D-optimal designs is available in the PI's forthcoming article in the Journal of Marketing Research (JMR). A thorough presentation and discussion of C-optimality is presented in the PI's book chapter forthcoming in Christie and Midmore (editors).
The PI also briefly investigated the question of how the number of alternatives in a choice set affect estimation efficiency. This turns out to be a complicated question that is directly tied to the specific attribute levels used in the different alternatives. In a Monte Carlo study, the PI found that adding alternatives with intermediate attribute levels can actually decrease the statistical information provided by a choice set. Exactly when and how this occurs would be an interesting research question to pursue further.
The second phase of the project derived optimal designs for models that include substitution effects or higher-order terms. Unfortunately, the PI found that a general optimal design solution for these non-linear models is extremely complex and probably impossible to obtain in theoretical terms. This is consistent with other findings in the optimal design literature. Exact optimal designs would depend, in a complicated way, on model parameters. The PI did, however, find two special cases where optimal designs can be specified. Both cases fit nicely into the known body of optimal design results.
The first case is a quadratic model where at least one attribute appears only linearly. In other words, though the full model has higher-order terms, it is linear with respect to at least one particular attribute. By focusing on this linear term, optimal designs can be obtained that mimic those for the standard, linear model.
Essentially, the results provided in the JMR article can be applied directly to this case, with one important change. The JMR results apply to linear (or main-effects) models. D-optimal choice set designs assign two levels to each attribute. For all attributes but one, these levels are the researcher's judgment of reasonable upper and lower bounds. The attributes are allocated according to the standard main-effects design arrays.
The level of the final attribute is manipulated to obtain certain response probability splits. For example, with a two-attribute choice set, the researcher should strive to obtain a .82/.18 response split between the two alternatives. With eight attributes, a .67/.33 split is optimal. The optimal split appears to move inward (closer to .50/.50) as the number of attributes increases.
With the quadratic model with a linear attribute, the important change from these results is that a main-effects design array is not sufficient for model identification. Instead, the researcher must use three-levels for the quadratic attributes. The design array becomes larger. The optimal choice split is essentially the same as in the main-effects case, except that it is based on the total number of variables (linear plus quadratic) in the model rather than the number of attributes.
The second case where specific design solutions are obtainable is the fully specified interactive model. This model includes all interactive terms. For example, with two attributes, the model includes two linear terms (say, x1 and x2) and one interactive term (x1x2). With three attributes, the model contains the following variables: x1, x2, x3, x1x2, x2x3, x1x3, and x1x2x3. These models require the full factorial design array (the full set of attribute permutations) for model identification.
The optimal response split for this case is intriguing. It always requires a .82/.18 probability split and is independent of the number of attributes in the model. Recall that the .82/.18 split is optimal for the two-attribute, main-effects model and that larger models have different optimal splits.
Results for the quadratic and interactive models are presented in "Optimal Design for Binary Choice Experiments with Quadratic or Interactive Terms."
How does one implement the above optimal designs in practice? The PI recommends a sequential approach. The researcher would start with a choice set design based on standard design arrays and specify levels for the manipulator attribute to obtain optimal response splits based on his or her intuition. Then, as choice responses are collected, the researcher can update the manipulator attribute levels to move responses toward the optimal ones.
The PI assisted researchers at Michigan State University in the first example of such an empirical study. The research team found that the process went smoothly and estimation results were more efficient than they otherwise would have been. Results of this study are provided in Steffens, Lupi, Kanninen, and Hoehn (2002).
Conclusions:
For main-effects and interactive models, all attributes but one should be assigned only two levels. These levels should be what the researcher judges to be reasonable upper and lower bounds. For quadratic models, attributes that have quadratic terms should be assigned three levels at their upper and lower bounds and the mid-point in between.
These attributes should be allocated across choice sets according to the relevant design arrays. For main-effects models, these design arrays are the standard, main-effects designs; for quadratic models, three-level designs; and for interactive models, full factorial design arrays.
Note that over-specifying interactive terms at the design stage-using a full factorial when only a main-effects design is ultimately needed-will not affect estimation efficiency. Using a quadratic specification when only main-effects will ultimately be estimated, however, decreases efficiency because the mid-point attribute level is not very informative for main-effects estimation.
Once all attributes but one are set, the researcher uses the final attribute to manipulate response rates. For fully specified interactive models, the optimal probability split is always .82/.18. This also is the optimal split for a model with two variables.
These optimal splits move inward as the number of columns in the data matrix increases. For example, the optimal split is .67/.33 for a model with eight columns in the data matrix.
Note that in practical situations, this difference is not so great. To obtain optimal response rates, the researcher will have to start with a prior guess for the manipulator attribute levels. Over the course of the experiment, the researcher can adjust the manipulator attribute to move responses toward their optimal splits. When the researcher does not know exactly how many attributes or combinations of attributes will ultimately be used in the model, he or she can focus on keeping the design within the bounds of .82-.67/.18-.33.
Many researchers would be uncomfortable applying these optimal designs directly because of the limited number of attribute levels used. For main-effects models, all attributes take only two levels. For quadratic terms, attributes take three levels. Keep in mind, however, that once a functional form is specified, these numbers are all that are required to estimate the model. In some sense, additional data points are superfluous. Either way, the most influential data points collected will dominate estimation. For this reason alone, it is advisable to focus on points of high information. Generally, these are the most extreme points with which a researcher is comfortable.
Note that extreme does not mean the far reaches of an attribute's range. A daily pass at a state park, for example, should not be offered at $100 (in today's dollars). With a main-effects model, the upper bound should be the maximum value thought to be on the linear portion of the utility model. This would be a much lower value.
Often in valuation research, sample sizes are small because of small budgets. This is where optimal design of experiments is most important. In these cases, the researcher must focus on high information points to get significant results, especially when there are many attributes.
More empirical research on this approach is needed. Small-scale is fine. Research questions abound and include the question of which attributes would best serve as manipulators, how often is it best to update the manipulator attribute, and how much statistical improvement can be obtained under controlled experimental conditions.
Journal Articles on this Report : 1 Displayed | Download in RIS Format
Other project views: | All 5 publications | 2 publications in selected types | All 1 journal articles |
---|
Type | Citation | ||
---|---|---|---|
|
Kanninen BJ. Optimal design for multinomial choice experiments. Journal of Marketing Research 2002;39(2):214-227. |
R827987 (2000) R827987 (Final) |
Exit |
Supplemental Keywords:
onjoint analysis, nonmarket valuation, contingent valuation, survey, willingness to pay, multinomial logit, D-optimality, C-optimality., RFA, Economic, Social, & Behavioral Science Research Program, Scientific Discipline, Economics, decision-making, Ecological Risk Assessment, Ecology and Ecosystems, Social Science, Economics & Decision Making, contingent valuation, multi-objective decision making, policy analysis, surveys, optimal experimentation, dichotomous-choice, decision analysis, nonmarket choice, market valuation models, non-market valuation, standards of value, D-optimality, public policy, willingness to pay, multinominal nonmarket choice experiments, conjoint analysisProgress and Final Reports:
Original AbstractThe perspectives, information and conclusions conveyed in research project abstracts, progress reports, final reports, journal abstracts and journal publications convey the viewpoints of the principal investigator and may not represent the views and policies of ORD and EPA. Conclusions drawn by the principal investigators have not been reviewed by the Agency.