You are here:
Variability in in vivo studies: Defining the upper limit of performance for predictions of systemic effect levels
Citation:
Pham, L., S. Watford, P. Pradeep, M. Martin, R. Thomas, R. Judson, Woodrow Setzer, AND K. Paul-Friedman. Variability in in vivo studies: Defining the upper limit of performance for predictions of systemic effect levels. Computational Toxicology. Elsevier B.V., Amsterdam, Netherlands, 15(August 2020):100126, (2020). https://doi.org/10.1016/j.comtox.2020.100126
Impact/Purpose:
Importantly, this work sets a threshold on the accuracy expected from predictive modeling of systemic effect points-of-departure and may help inform fit-for-purpose acceptance criteria for systemic toxicity points-of-departure derived from NAMs. In the present work, statistical approaches were used to estimate the variance in systemic effect levels from in vivo studies of adult animals described in the publicly-available Toxicity Reference Database (ToxRefDB) version 2.0. In doing so, we addressed two primary questions regarding these reference data for systemic effects: (1) for a given chemical, what is an estimate of a bound on accuracy for new approach methodology (NAM) prediction of systemic toxicity given the variability in effect levels from in vivo reference data; and, (2) what is an estimate of the upper limit on the performance of a NAM based on available reference data, i.e. what is the maximum amount of variance that a NAM can explain in this reference data set? Herein we estimate the size of a “minimum prediction interval,” i.e. the interval in which a new observation or model prediction would be expected to fall; this minimum prediction interval is based on variance estimates of systemic toxicity information in ToxRefDB, and does not reflect variance that would likely be contributed by a less than perfect NAM or predictive model.
Description:
New approach methodologies (NAMs) for hazard are often evaluated via comparison to animal studies; however, variability in animal study data limits NAM accuracy. The US EPA Toxicity Reference Database (ToxRefDB) enables consideration of variability in effect levels, including the lowest effect level (LEL) for a treatment-related effect and the lowest observable adverse effect level (LOAEL) defined by expert review, from subacute, subchronic, chronic, multi-generation reproductive, and developmental toxicity studies. The objectives of this work were to quantify the variance within systemic LEL and LOAEL values, defined as potency values for effects in adult or parental animals only, and to estimate the upper limit of NAM prediction accuracy. Multiple linear regression (MLR) and augmented cell means (ACM) models were used to quantify the total variance, and the fraction of variance in systemic LEL and LOAEL values explained by available study descriptors (e.g., administration route, study type). The MLR approach considered each study descriptor as an independent contributor to variance, whereas the ACM approach combined categorical descriptors into cells to define replicates. Using these approaches, total variance in systemic LEL and LOAEL values (in log10-mg/kg/day units) ranged from 0.74 to 0.92. Unexplained variance in LEL and LOAEL values, approximated by the residual mean square error (MSE), ranged from 0.20-0.39. Considering subchronic, chronic, or developmental study designs separately resulted in similar values. Based on the relationship between MSE and R-squared for goodness-of-fit, the maximal R-squared may approach 55 to 73% for a predictive model of systemic toxicity using these data as reference. The root mean square error (RMSE) ranged from 0.47 to 0.63 log10-mg/kg/day, depending on dataset and regression approach, suggesting that a two-sided minimum prediction interval for systemic effect levels may have a width of 58 to 284-fold. These findings suggest data-driven considerations for building scientific confidence in NAM-based systemic toxicity predictions.
URLs/Downloads:
DOI: Variability in in vivo studies: Defining the upper limit of performance for predictions of systemic effect levels