cc: Edward Cook <drdendro@ldeo.columbia.edu>, Ned Guttman <Ned.Guttman@noaa.gov>, potter4@bellsouth.net, Keith Briffa <k.briffa@uea.ac.uk>, "Klein Tank, Albert" <Albert.Klein.Tank@knmi.nl>, Richard Heim <Richard.Heim@noaa.gov>
date: Mon, 4 Jun 2007 08:17:25 -0400
from: Edward Cook <drdendro@ldeo.columbia.edu>
subject: Re: scPDSI program
to: Gerard van der Schrier <schrier@knmi.nl>

   Hi Gerard,

   Thanks for the email. The choice of the calibration period on the PDSI estimates outside
   that period is something that probably affects all versions of the PDSI. The SPI ought to
   be similarly affected if the quantiles are based on a pdf fitted to a sub-period of data.
   That being the case, unless the calibration period is a truly unbiased expression of
   longer-term variability, I am not sure what can be done about it. I also agree that one
   should use the longest calibration period possible, but globally it is not really feasible
   to start before, say, 1950 over large areas of Asia. So this will make pre-1950 PDSI
   variability strongly conditional on climate statistics of the calibration period.

   Cheers,

   Ed
   ==================================
   Dr. Edward R. Cook
   Doherty Senior Scholar and
   Director, Tree-Ring Laboratory
   Lamont-Doherty Earth Observatory
   Palisades, New York 10964  USA
   Email: [1]drdendro@ldeo.columbia.edu
   Phone: 845-365-8618
   Fax: 845-365-8152
   ==================================
   On Jun 4, 2007, at 5:47 AM, Gerard van der Schrier wrote:

   Dear Ned, Ed,

   Thanks for your emails. I must admit that I haven't had the chance to

   work with the sc-pdsi program lately (=since 1 year or so......). The

   plans are to start working on this again in next autumn/winter.

   About the sensitivity to the length of the calibration period: The

   self-calibrating aspect makes that ca. 2% of the available months in the

   calibration period are in the "extemely dry" category (and 2% will be in

   the "extremely wet" category). The 2% threshold is more-or-less

   abritrary. The parameters in the PDSI-calculation are set to satisfy

   this constraint.

   When the calibration period does not cover the complete period over

   which the sc-pdsi is calculated, then you can expect PDSI values

   (hugely) outside the [-4, 4] range. Especially when a dry (or wet)

   period happens to fall outside the calibration period. I guess this was

   the reason why Wells et al. in the paper in which they introduce the

   sc-pdsi, advise to make the calibration period as long as possible.

   Note that this leaves another problem of the PDSI untouched: the median

   of a distribution of sc-pdsi values is not guaranteed to be zero. This

   remains unnoticed when a probability distribution is calculated over

   many different timeseries, but it can be seen in individual timeseries

   (e.g. fig. 4 of Wells et al. (2004), J. Clim. vol. 17, pp. 2335-2351).

   Obviously, I also included a spin-up period of 10 years for the

   waterbalance model, using climatological data.

   There are indeed problems with the sc-pdsi which need to be looked at

   before it is ready for an operational application.  Nevertheless, we

   think it compares favourably to the "traditional" PDSI.

   Our plans are to calculate sc-pdsi for the updated global temperature

   and precipitation datasets of the Climatic Research Unit. There are some

   vague plans to use the index in a more operational way, and apply it to

   a network of European data which are continuously updated. Obviously,

   much needs to be done before this works satisfactory.

   Ned, could I get back to you when I actually start working on this?

   Many thanks for your thoughts.

   Best Regards,

   Gerard

   ----------------------------------------------------------

   Gerard van der Schrier

   Royal Netherlands Meteorological Institute (KNMI)

   dept. KS/CK

   PO Box 201

   3730 AE De Bilt

   The Netherlands

   [2]schrier@knmi.nl

   +31-30-2206597

   www.knmi.nl/~schrier

   -----------------------------------------------------

   Ed, Gerhard,

      After looking at the structure of Gerhard's code (module oriented)

   and Richard's code (hardwired for I/O), I decided that the easiest way

   to write something that we can use at NCDC would be to incorporate

   Gerhard's subroutines into Richard's program.  The result is attached.

   I seem to remember that there may have been a sign problem in

   Gerhard's code, but I am not sure now.  I did this a month or two ago,

   and then started getting ready for retirement (tomorrow is my last

   day).  I did check out Gerhard's code with Palmer's and Nate Wells'

   papers to make sure the code truly reflects both Palmer's and Wells'

   logic.  I did write a simple subroutine to calculate the regressions

   rather than use the sophisticated library subroutine that Gerhard

   used.  The regression results from the two methods are close but not

   identical.

      Then I ran the attached program on US Climate Division data

   (1895-2006).  The output was not satisfactory - I was getting

   outrageous PDSI values in some divisions for some periods.  The

   calibration period for calculating the CAFEC values was 1950-2000, the

   period that is used for the North American Drought Monitor (Mexico

   does not have long term data).  As a test, I reran the program using a

   calibration period of (I think) 1900-2000 and the results were much

   better.  After thinking about it for a while, but not a long time, it

   is logical that the PDSI is sensitive to the calibration period, since

   the components of the water balance at the start of calibration period

   reflect existing and immediately preceding conditions such as droughts

   and wet spells.  This is likely skewing the output of the sc-pdi in an

   unsatisfactory manner.

      I have not pursued this, and would hope to follow through with it

   somehow with contract work after tomorrow, but it is an issue that

   needs to be resolved.  I also think the sc-pdi needs to be thoroughly

   tested before it is used in any kind of operational manner.  There are

   too many things in the process that can go wrong when programmed.  I

   also think that the characteristics of the sc-pdi need to be evaluated

   and affirmed.  For example, is the method truly spatially invariant?

      Let me give you an email address where you can reach me after

   tomorrow (or maybe today since I don't know when all my computer

   accounts will be killed):

   [3]potter4@bellsouth.net

      Bottom line - I would not use the sc-pdi now.  It needs much more

   review.

   Cheers,

   --

   ----------------------------------------------------------

   Gerard van der Schrier

   Royal Netherlands Meteorological Institute (KNMI)

   dept. KS/CK

   PO Box 201

   3730 AE De Bilt

   The Netherlands

   [4]schrier@knmi.nl

   +31-30-2206597

   www.knmi.nl/~schrier

   ----------------------------------------------------------

