date: Wed, 04 Jul 2007 16:21:48 +0100
from: david.parker@metoffice.gov.uk
subject: Re: Fwd: RE: IPCC Table 3.2
to: "Jones, Phil" <p.jones@uea.ac.uk>

Phil

Thanks for both. Steve McIntyre has ashed for the software I used for
calculating the DW statistic so I will send him just the relevant lines.

I use a printout of the Chapter 3 pdf!

Regards

David

On Wed, 2007-07-04 at 14:03 +0100, Phil Jones wrote:
>   David,
>       Thanks. Tim thinks he could get the procedures to work,
>   but won't bother as your explanation is fine. I'll paraphrase it
>   and send off.  I doubt though that he will be happy. He will
>   probably say we didn't give enough explanation. I'll
>   be cc'ing to the same list as earlier.
> 
>     Thanks again.
> 
>     It would be nice to get the CUP book. I keep opening the pdf
>   version of the chapter!
> 
>   Cheers
>   Phil
> 
> 
> At 12:06 04/07/2007, you wrote:
> >Phil
> >
> >The DW statistic was done on the residuals after removing the AR1
> >persistence as modelled by the restricted maximum likelihood software.
> >That is why the values were close to 2. This procedure is correct
> >because the restricted maximum likelihood software widens the error-bars
> >to take account of AR1; the DW is a test to see whether any further
> >widening is needed, and the results show that it isn't. Steve McIntyre
> >probably used the residuals unadjusted for AR1.
> >
> >I attach our software and 2 sample series. I don't think "R" is used
> >widely here. The DW coding is at lines 78-91 of the plot routine
> >(p_plot_glm_out_annual.pro).
> >
> >I hope this helps
> >
> >Regards
> >
> >David
> >
> >
> >On Wed, 2007-07-04 at 09:08 +0100, Phil Jones wrote:
> > >
> > >  David,
> > >    Any idea what McIntyre is doing wrong? Maybe check the pv-wave
> > >  code and if possible send to me. It is a bit like IDL so I can
> > >  probably follow. Maybe there is a routine in PV-Wave for DW.
> > >    I've calculated DW on numerous occasions over the past 30 years
> > >  for the following:
> > >
> > >  1. In the riverflow reconstruction work. Here I was reconstructing
> > > monthly
> > >  flows from rainfall. In the southeastern parts of Britain where there
> > > is
> > >  a high groundwater component, rainfall errors propagate for up to a
> > > year.
> > >  Here I got some low DWs with values between 0.5 and 1.5.
> > >
> > >  2. In reconstruction from paleo data we calculated DW and
> > > occasionally
> > >  got values down to the range 1.2 to 1.5.
> > >
> > >  90% of the time the values were between 1.8 and 2.4 which is where
> > > ours
> > >  are in the Tables.
> > >
> > >  I've never been able to get values down to the 0.27 and 0.49 that
> > > McIntyre
> > >  is talking about.
> > >
> > >  Tim Osborn thinks he would be able to run your pv-wave script here,
> > > so
> > >  if you can send this with one or two of the series - need the
> > > residuals,
> > >  we can check things out.
> > >
> > >  Also looking at his R script, I don't think DW can have a p-value as
> > > such,
> > >  as it has the two critical levels I was talking about in my response.
> > >
> > >  What I think you have calculated DW based upon is the residuals,
> > >  so the difference between the original series and the fitted line.
> > >
> > >  If anyone at MOHC is adept at R, it would be very nice to point out
> > > what is
> > >  wrong with his script !!!
> > >
> > >  Cheers
> > >  Phil
> > >
> > >
> > >
> > > > X-YMail-OSG:
> > > > Lbmzx_0VM1nx78zXgGstXLvBNj_1bKKWu3u3EfY0YVl7XZyftROTuk6Cnr.ZURgyAw--
> > > > From: "Steve McIntyre" <stephen.mcintyre@utoronto.ca>
> > > > To: "'Phil Jones'" <p.jones@uea.ac.uk>
> > > > Subject: RE: IPCC Table 3.2
> > > > Date: Tue, 3 Jul 2007 13:03:18 -0400
> > > > X-Mailer: Microsoft Outlook, Build 10.0.2627
> > > > X-UEA-Spam-Score: 0.1
> > > > X-UEA-Spam-Level: /
> > > > X-UEA-Spam-Flag: NO
> > > >
> > > > Dear Phil, thanks for the prompt reply, but there are a number of
> > > > points that remain very unclear.  I am extremely familiar with the
> > > > Durbin-Watson statistic as it is familiar to all econometricians. I
> > > > use the R language which has a convenient Durbin-Watson test in the
> > > > lmtest package (the dwtest function.) When I ran a Durbin-Watson
> > > > test on residuals from fitting a trend, I obtained a Durbin-Watson
> > > > statistic of 0.49 for an OLS-fitted trend to the HadSST2 series
> > > > presently online (over 1850-2005).When I re-fitted a trend line
> > > > using the reported slope of 0.038 deg C/decade, I obtained an even
> > > > lower Durbin-Watson statistic of 0.27.   For this same situation,
> > > > you reported a DW statistic of 2.2.    I've attached a script in R.
> > > > You say that"We used the lag-1 autocorrelations to calculate the
> > > > reduced number of degrees of freedom of the residuals."   In order
> > > > to get a Durbin-Watson statistic of 2.2, you must have done
> > > > something to the data that is not a typical procedure and which is
> > > > not explained in Diggle.  The best guess that I could come up with
> > > > as to what you did was that you might have fitted an AR1 arima model
> > > > to the trend residuals and then calculated a DW statistic for the
> > > > residuals for the arima-fit. However, this is just speculation and
> > > > there is no clue in AR4 as to what was done.
> > > >
> > > > BTW the usual interpretation of the DW test in econometrics is as a
> > > > test for first-order autocorrelation, so the exact meaning of using
> > > > a DW test "after allowing for first-order serial correlation" is by
> > > > no means obvious.  Again, if you can direct me to an article
> > > > describing the exact procedure that you used together with its
> > > > statistical properties, I'd appreciate it.
> > > >
> > > > Regards, Steve McIntyre
> > > >
> > > >
> > > >
> > > >         -----Original Message-----
> > > >         From: Phil Jones [mailto:p.jones@uea.ac.uk]
> > > >         Sent: Monday, July 02, 2007 9:59 AM
> > > >         To: Steve McIntyre
> > > >         Cc: Palmer Dave Mr (LIB); david.parker@metoffice.gov.uk;
> > > >         Kevin Trenberth; Susan Solomon; Martin Manning
> > > >         Subject: Re: IPCC Table 3.2
> > > >
> > > >
> > > >          Steve,
> > > >             The Durbin-Watson statistics were in an earlier draft of
> > > >         the chapter. They were
> > > >          removed simply for space reasons, as none were significant.
> > > >         As you can see,
> > > >          we also removed the lag-1 autcorrelations as well.
> > > >
> > > >             REML comes from Diggle et al 1999 (section 4.5 pp
> > > >         64-68). This reference is
> > > >          given at the end of the chapter. The page numbers refer to
> > > >         the 1999 edition of the book.
> > > >          There is a later one available on Amazon, so the page
> > > >         numbers may differ in that
> > > >          edition. David Parker programmed the calculations of all
> > > >         the trends. As far as I
> > > >          know he didn't do this with any specific statistical
> > > >         packages. He likely used PV-WAVE
> > > >          which the Hadley Centre used for almost all their analysis
> > > >         work. The use of REML
> > > >          is discussed in Appendix 3.A.
> > > >
> > > >           DW is very simple to calculate. We used the lag-1
> > > >         autocorrelations to calculate the
> > > >          reduced number of degrees of freedom of the residuals. This
> > > >         number was used with
> > > >          the DW statistic to estimate the significance. Basically,
> > > >         any DW value above
> > > >          about 1.8 is not significant. DW Tables are in some
> > > >         statistics books. There should
> > > >          be two significance values (for any DW value and N, here
> > > >         the effective number
> > > >          of degrees of freedom). For the lower of these, values
> > > >         below would be significant.
> > > >          Values above the upper are not significant. For values in
> > > >         between nothing can
> > > >          be said.  We were always above the upper value. For random
> > > >         numbers, the
> > > >          DW statistic should return a value about 2. There are
> > > >         different sets of Tables for
> > > >          different significance levels (1%, 5% etc). We used 5%,
> > > >         which is generally
> > > >          the one given in text books.
> > > >
> > > >            Any statistical package would likely not use the reduced
> > > >         number of degrees
> > > >          of freedom (reduced based on the lag-1 autocorrelation)
> > > >         when giving the
> > > >          significance of DW. Using the reduction to the degrees of
> > > >         freedom makes the
> > > >          test harder to pass.
> > > >
> > > >          Phil
> > > >
> > > >
> > > >         At 17:36 29/06/2007, Steve McIntyre wrote:
> > > >         Dear Phil,
> > > >         In Table 3.2 of IPCC AR4, you refer to Durbin-Watson
> > > >         statistics for various trend calculations, but do not show
> > > >         them. Could you please provide me with these statistics.
> > > >
> > > >         I am unfamiliar with any prior use of the Durbin-Watson
> > > >         statistic after allowing for first-order serial
> > > >         correlation. Could you please provide me your statistical
> > > >         reference showing how one calculates a Durbin-Watson
> > > >         statistic after allowing for first-order serial
> > > >         correlation and giving significance levels for the
> > > >         statistic after allowing for first-order serial
> > > >         correlation.
> > > >
> > > >         Could you please identify the statistical packages used in
> > > >         your calculation of REML trends and Durbin-Watson
> > > >         statistics?
> > > >
> > > >         Would it be correct to say that (1) fitted a trend to the
> > > >         various series; (2) fitted an AR1 arima model to the
> > > >         residuals from (1)? (3) carried out a Durbin-Watson test on
> > > >         the residuals from (2)?
> > > >
> > > >         Where applicable, these requests are made under FOI
> > > >         provisions.
> > > >
> > > >         Thank you for your attention, Steve McIntyre
> > > >
> > > >
> > >
> > >
> > >         Prof. Phil Jones
> > >         Climatic Research Unit        Telephone +44 (0) 1603 592090
> > >         School of Environmental Sciences    Fax +44 (0) 1603 507784
> > >         University of East Anglia
> > >         Norwich                          Email    p.jones@uea.ac.uk
> > >         NR4 7TJ
> > >         UK
> > > 
> > ---------------------------------------------------------------------------- 
> >
> > >
> >
> >Prof. Phil Jones
> >Climatic Research Unit        Telephone +44 (0) 1603 592090
> >School of Environmental Sciences    Fax +44 (0) 1603 507784
> >University of East Anglia
> >Norwich                          Email    p.jones@uea.ac.uk
> >NR4 7TJ
> >UK
> >---------------------------------------------------------------------------- 
> >
> >
> >--
> >David Parker Met Office Hadley Centre FitzRoy Road  EXETER  EX1 3PB  UK
> >E-mail: david.parker@metoffice.gov.uk
> >Tel: +44-1392-886649     Fax: +44-1392-885681     http://www.metoffice.gov.uk
> >
> >
> >
> >
> >
> >
> >
> >
> >
> 
> Prof. Phil Jones
> Climatic Research Unit        Telephone +44 (0) 1603 592090
> School of Environmental Sciences    Fax +44 (0) 1603 507784
> University of East Anglia
> Norwich                          Email    p.jones@uea.ac.uk
> NR4 7TJ
> UK 
> ----------------------------------------------------------------------------                                                                                 
-- 
David Parker Met Office Hadley Centre FitzRoy Road  EXETER  EX1 3PB  UK
E-mail: david.parker@metoffice.gov.uk
Tel: +44-1392-886649     Fax: +44-1392-885681     http://www.metoffice.gov.uk

