cc: Edward Cook <drdendro@ldeo.columbia.edu>
date: Fri, 13 Jun 2008 12:08:32 -0400
from: Edward Cook <drdendro@ldeo.columbia.edu>
subject: Fwd: Some PR Challenge comments
to: Keith Briffa <K.Briffa@uea.ac.uk>

   Just what I sent Casper. FYI only.

   ==================================
   Dr. Edward R. Cook
   Doherty Senior Scholar and
   Director, Tree-Ring Laboratory
   Lamont-Doherty Earth Observatory
   Palisades, New York 10964  USA
   Email: [1]drdendro@ldeo.columbia.edu
   Phone: 845-365-8618
   Fax: 845-365-8152
   ==================================

   Begin forwarded message:

   From: Edward Cook <[2]drdendro@ldeo.columbia.edu>
   Date: June 13, 2008 10:46:43 AM EDT
   To: Caspar Ammann <[3]ammann@ucar.edu>
   Cc: Edward Cook <[4]drdendro@ldeo.columbia.edu>, Nicholas Graham <[5]ngraham@hrc-lab.org>,
   "Rosanne D'Arrigo" <[6]druidrd@ldeo.columbia.edu>
   Subject: Some PR Challenge comments

   Hi Casper,
   As you are aware, I have some serious concerns about how the PR Challenge has been
   formulated. So as of now, I have no intention of participating in it. I mentioned to you in
   Trieste that I thought the better way to go was to use real climate data and real proxies
   for a range of reconstruction problems (temperature, precipitation, drought, pressure on
   local, regional, hemispheric scales) such that TRUE uncertainties in the reconstructions
   and the methods used could be objectively and realistically evaluated. The work Keith,
   Phil, and I did for NATO is a very useful template for doing so and the results can be
   objectively evaluated by setting up the testing and comparison procedures in an a priori
   way. However, I also recognize the value of pseudo-proxies in this context and support the
   original PR Challenge idea of using model output to produce pseudo-climate and pseudo-proxy
   data for testing as well. Indeed, one could create the pseudo-data sets to match fairly
   closely the properties of the actual data sets chosen for testing. Including both real and
   pseudo data in the experiments would dramatically strengthen your PR Challenge in my
   opinion because it would reduce the dependence of the results and their interpretations on
   the considerable uncertainty that still remains in how to generate pseudo-proxy data using
   forward-modeling methods. Tree rings are most developed here for forward modeling I think
   (e.g. the Vaganov-Shashkin model), but to assume that the V-S model can magically model the
   tremendous complexity of growth responses to climate in tree rings at the genus and species
   levels is simply wrong. Maybe the V-S model will get a lot of it right, but it will likely
   be in a highly idealized way that will very likely under-estimate the true uncertainty in
   tree-ring reconstructions. I am also not sure that useful forward models for other proxies
   like corals even exist yet.
   Perhaps the biggest complaint I have with the PR Challenge is its "double-blind" design
   that gives the tester no information on the problem being tested even though no such
   situation ever occurs in practice. At that level alone, the "double-blind" design used in
   the PR Challenge is very odd. Regardless, I fully understand the value of "double-blind"
   statistical experiments as the only way to test for true causality in, for example, cancer
   drug tests. But this is simply not the kind of problem we are dealing with in the proxy
   reconstruction game (with all due respect to your statistician colleagues). We are much
   more involved in something more akin to epidemiological hypothesis testing on real
   pre-existing data, like the association between smoking and lung cancer in humans. Do
   medical epidemiologists perform "double-blind" experiments to determine if smoking causes
   lung cancer? Basically no. They use the data available to them to answer that hypothesis
   with a high degree of statistical certainty without the need to conduct "double-blind"
   experiments on humans (not possible for smoking and lung cancer in any case). So in my
   opinion, the "double-blind" approach is unnecessary and, I would argue, even inappropriate
   here. I also don't like it because that is not the way we should be conducting our science.
   When I worked with Keith and Phil for a month at CRU on our NATO test data sets and
   reconstruction methods, we worked in a completely objective and open way that enabled us to
   debate various options and interpretations of our programs and reconstructions. We were
   also able to thoroughly test our programs using identical data for the special case of full
   MLR. This enabled us to be absolutely certain that we could compare results in the various
   best-subset cases that are typically used for reconstructing climate from tree rings. There
   does not appear to be any such mechanism for doing so in your PR Challenge, so differences
   found may be as much related to odd programming matters rather than the methods being
   compared. In any case, comparison of reconstruction methods should be conducted openly and
   that is the way that I love to work. So if the "double-blind" requirement were removed from
   the PR Challenge, I would be much more inclined to participate. I am not adverse to being
   proven wrong (or "less right" perhaps) in an open environment that allows for direct
   "give-and-take" on the merits of the cases being tested and argued. A "double-blind"
   approach is explicitly designed to eliminate the scientific openness needed to evaluate and
   debate reconstruction methods in my opinion. At that level, and for reasons stated above, I
   reject it.
   Regards,
   Ed
   ==================================
   Dr. Edward R. Cook
   Doherty Senior Scholar and
   Director, Tree-Ring Laboratory
   Lamont-Doherty Earth Observatory
   Palisades, New York 10964  USA
   Email: [7]drdendro@ldeo.columbia.edu
   Phone: 845-365-8618
   Fax: 845-365-8152
   ==================================

