You are here:
Response to Gelman Comments in Chance Magazine
Citation:
BLACKMAN, C. F. Response to Gelman Comments in Chance Magazine. JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION. American Statistical Association, Alexandria, VA, 25(1):62-63, (2012).
Impact/Purpose:
This Letter to the Editor was written to clarify and respond to negative statements made about EPA scientists and research. The Letter to the Editor will correct misunderstandings created by an article in Chance magazine and, in so doing, provide an opportunity to highlight the creative and innovative science done by EPA and its impact on a particular scientific field.
Description:
Carl Blackman Letter to the Editor of Chance Magazine: I appreciate being given the opportunity to comment on an editorial column by Andrew Gelman, entitled "Ethics and Statistics", that appeared in 2011, volume 24, no. 4, pages 51-53, in which my colleagues and I and our 1988 publication were prominently featured. Let me state from the outset that I fully support the principles of data sharing, and open and transparent exchange of scientific information, as well as the scientific publication and peer review process which supports open criticism, independent reproducibility, and scrutiny of scientific results. Furthermore, I believe that I have been a faithful adherent to those principles throughout my 40+ year career as a researcher at the Environmental Protection Agency, with many of my most useful research products and contributions resulting from open collaborations with a great many scientists. Dr. Gelman's column purported to present some important issues in scientific ethics from a statistician's viewpoint, elaborating on an experience from his graduate school years to illustrate his point. He begins his article with the statement: "An ethics problem arises when you are considering an action that (a) benefits you or some cause you support, (b) hurts or reduces benefits to others, and (c) violates some rule." Dr. Gelman then proceeds anecdotally to relate a distantly recalled experience, during the course of which he dismisses published, peer-reviewed scientific results of over 20 years ago, and impugns the scientific competency, integrity, and motives of me and my experienced EPA statistician coworker. I would counter that the unprofessional and self-serving manner in which this article was written, misrepresenting the original personal interaction, and ignorant of (and uninterested in) the biophysical complexities and underpinnings of this and careful follow-up studies (that have been independently reproduced and considered influential), should be considered beneath the standards of this magazine and, ironically, would seem to violate the very ethical principles that Dr. Gelman espouses. Dr. Gelman levels the charge, 20 years after the fact, that I violated the principle of openness in scientific research by denying his request to send him copies of my logbooks, and that I designed experiments and data analyses that led to a "waste of effort," presumably because I and my coworker misapplied statistical principles in the analysis of the experimental findings. Both assertions are based on misleading and incomplete information, and in my view, are groundless. I indeed recall receiving Gelman's letter in 1988 or 1989 requesting data from our publication (Blackman et al. 1988). The letter addressed a statistical issue, so I discussed the request with my co-author, an experienced and practiced biostatistician. The data in question had been collected in 1981-1983, nearly 6 years previously, and by 1984 we had tested and confirmed the fundamental hypothesis suggested from the original P-value plot, vlz., that the local static magnetic field was involved in determining which frequencies could be biologically effective (Blackman et al., 1985). Thus, in 1988 I concluded that it was unlikely that further statistical examination of the data that was used to generate the original P-value plots would be Ltr to Editor of Chanceregarding Gelman (final)+l.docx • • Page2 of4 profitable in advancing the understanding of that science. When Gelman called me to repeat his request} I inquired about his reason for wanting to use these data and he indicated that he wished to test some statistical hypotheses. When I offered to describe some of the sources of variance we had discovered in the 10 years during which we had refined the test} he appeared uninterested. As we talked} I inferred that his research status at Harvard was likely to be a graduate student and I recall having the strong impression that he was searching for data that he could work with in his spare time. It was disconcerting to me that he was not forthcoming about his status or motives (was it in fact ethical to "be careful not to identify myself as a PhD studenr'P), and that he expressed no interest in learning of the larger context or implicit constraints in the experimental design and data analysis that were not obvious in the paper} but elaborated in earlier studies. I made the decision at that point that the time and effort I would have had to invest to locate and photocopy data scattered across many lab notebooks from 6 years earlier was excessive} since I believed that results of a statistical reanalysis of that data} whatever the outcome} would not advance the understanding of the science underpinning our results. I remember thinking that if he really were a graduate student} I should speak to his research advisor to determine the scientific and educational value of his working with these data. Since Gelman chose to not acknowledge that he was a graduate student at the time} I made a judgment that his efforts would not be mutually beneficial} and likely did not have the support or endorsement of his mentor. In fact} I was no longer performing studies in this research area because the EPA research program was discontinued in early 1986} and I had archived research materials from the 1976-1985 years. I was aggressively pursuing other experimental techniques and establishing collaborations that were completely outside the electromagnetic research area. Further} the size of the data set for this particular paper was very large and not easily assembled as it was scattered across two projects. There was another hurdle} viz., I was required to get permission from EPA to share the raw data} something I believe I could have done had I considered this to be a worthwhile pursuit or had I been able to indicate that I was collaborating with a Harvard professor and his student. Dr. Gelman admits that he chose to deliberately conceal his status at Harvard} for whatever reason. I would argue that it was this misrepresentation that prevented a more productive and mutually beneficial exchange} and unfortunately} he is correct that the raw data are no longer available. This electromagnetic research effort was a basic research project for which the data had to be retained in accordance with the EPA records retention schedule; they were destroyed after that date. In view of these facts} I assert that my refusal of the data request was not unethical and should not have been characterized as such. Dr. Gelman also asserts that I designed experiments that were inadequate and wasteful of resources} and that I misused the concept of statistical significance. I do not intend to defend the classical model of statistical analysis against the newer model that he is championing; we had a more critical audience to convince in the early and mid 1980s: experimental biologists} electrical engineers and biophysicists. Thus} our experimental design was based on our reported findings that there were temperature changes in the samples during the 20-minute exposure period as the samples rose to the set-point temperature} and that temperature change could influence the endpoint (Blackman et al. 1982). We believed it was essential to Ltrto Editorof Chance regarding Gelman (final)+l.docx • Page30f4 ensure that the sham and exposed samples had the same temperature history; as we were not able to get a comparable exposure chamber to serve as a coincident sham control, we always paired the two treatments within a one-hour period. Thus, in Gelman's designation, samples Ai and Bl were subjected to the same temperature change conditions, and samples A2 and B2 were subjected to the same temperature conditions, but the temperature conditions for Ai & Bl were not the same as those for A2 & B2. I do not question Gelman's conclusions that on the average the results of samples A2, Bl and B2 were indistinguishable, but, in my considered judgment, the audience I was addressing would not have accepted a revised experimental protocol as he suggests. I believe our procedure was the best we could devise at the time, and it was indeed effective in satisfying many concerns of the target audience. Lastly, Dr. Gelman said of our work "They used this completely inappropriate statistical distinction as the basis of several pages of speculation. This ... led to a waste of effort." (page 52). We clearly stated in the paper that "To analyze the data further, we used a transformation, the P-value, which combines the difference between the means of the exposed and unexposed groups, as well as the variance of each group. This transformation may be useful for hypothesis generation, but not as proof of any underlying pattern(s)." We had collected the response data as a function of frequency in the early 1980s but did not immediately present it because the response profile was a tantalizing enigma. The speculative use of P-values to highlight features of the data was far from "a waste of effort"; rather, it led to the report by Blackman et al. (1985) demonstrating that the earth's magnetic field could influence which frequencies of magnetic fields could be effective. That report quickly led others in the USand abroad to confirm and extend the conditions for static magnetic-field controlled resonances. The research in this 1985 paper was undertaken because of the use of P-values, in the larger context of the biophysical considerations, to propose novel testable hypotheses. Its publication initiated a new direction in the electromagnetic research area by investigators in the US, USSR and Sweden; the publication now has over 200 citations, and has spawned a spirited debate regarding the underlying physical mechanism(s) of the action of electromagnetic fields on biological systems. In the 1988 paper under question, I merely provided the answer to the question, what aspect of the data caused us to think that the static magnetic field could be involved? The P-value inferences from that early publication directly led to scientific discovery that has had substantial, beneficial consequences for expanding the understanding of how electromagnetic fields can influence biological systems and processes. I understand that Dr. Gelman is waging a battle to help scientists avoid the subtle trap of drawing conclusions from differences between significance levels. A battle is more persuasively argued in black and white terms, so he conveniently ignores our published qualification in the use of P-values and followup studies building on these inferences. I do not wish to engage in this larger battle, but simply to point out that the use of P-values in the context of a complex biophysical problem with realworld considerations and influences led to a very productive outcome. I agree with Dr. Gelman that it is important for researchers in all areas of life sciences to utilize best practices in the application of statistical principles, and to be called to account when this is not the case. Ltrto Editor of Chance regarding Gelman (final)+1.docx • Page4 of4 However, I would add that it behooves statisticians, at all levels of education, to consider the full context of a study and to engage researchers in a constructive, respectful manner before leveling charges of incompetence and unethical behavior in areas of research about which they have limited understanding. Perhaps there are even good reasons why the statistically sophisticated neuroscience research community, in some cases, still draws conclusions from the differences between significance levels. Carl F. Blackman, PhD Integrated Systems Toxicology Division (B-105-03) National Health and Environmental Effects Laboratory US Environmental Protection Agency Research Triangle Park, NC 27711 USA Disclaimer: This letter to the editor has been subjected to review by the National Health and Environmental Effects Research Laboratory and approved for publication. Approval does not signify that the contents reflect the views of the Agency, nor does mention of trade names or commercial products constitute endorsement or recommendation for use. Further Reading Gelman, A. 2011. Open Data and Open Methods, Ethics and Statistics. Chance 24(4): 51-53. Blackman, C.F., S.G. Benane, L.S. Kinney, W.T. Joines, and D.E. House. 1982. Effects of ELF fields on calcium-ion efflux from brain tissue in vitro. Radiation Research 92(3): 510-520. Blackman, C.F., S.G. Benane, J.R. Rabinowitz, D.E. House, and W.T. Joines, W.T. 1985. A role for the magnetic field in the radiation-induced efflux of calcium ions from brain tissue in vitro. Bioelectromagnetics 6(4): 327-337. Blackman, C.F., S.G. Benane, D.J. Elliott, A.R. Wood, D.E. House, and M.M. Pollock. 1988 Influence of electromagnetic fields on the efflux of calcium ions from brain tissue in vitro: Three models consistent with the frequency response up to 510 Hz. Bioelectromagnetics 9(3):215227. Ltr