date: Thu, 10 Jul 2008 10:47:24 -0700
from: Ben Santer <santer1@llnl.gov>
subject: [Fwd: JOC-08-0098.R1 - Decision on Manuscript]
to: "Thorne, Peter" <peter.thorne@metoffice.gov.uk>, Leopold Haimberger <leopold.haimberger@univie.ac.at>, Karl Taylor <taylor13@llnl.gov>, Tom Wigley <wigley@cgd.ucar.edu>, John Lanzante <John.Lanzante@noaa.gov>, ssolomon@frii.com, Melissa Free <Melissa.Free@noaa.gov>, peter gleckler <gleckler1@llnl.gov>, "'Philip D. Jones'" <p.jones@uea.ac.uk>, Thomas R Karl <Thomas.R.Karl@noaa.gov>, Steve Klein <klein21@mail.llnl.gov>, carl mears <mears@remss.com>, Doug Nychka <nychka@ucar.edu>, Gavin Schmidt <gschmidt@giss.nasa.gov>, Steven Sherwood <Steven.Sherwood@yale.edu>, Frank Wentz <frank.wentz@remss.com>

<x-flowed>
Dear folks,

I just returned from my trip to Australia - I had a great time there. 
Now (sadly) it's back to the reality of Douglass et al. I'm forwarding 
the second set of comments from the two Reviewers. As you'll see, 
Reviewer 1 was very happy with the revisions we've made to the paper. 
Reviewer 2 was somewhat crankier. The good news is that the editor 
(Glenn McGregor) will not send the paper back to Reviewer 2, and is 
requesting only minor changes in response to the Reviewer's comments.

Once again, Reviewer 2 gets hung up on the issue of fitting higher-order 
autoregressive models to the temperature time series used in our paper. 
As noted in our response to the Reviewer, this is a relatively minor 
technical point. The main point is that we include an estimate of the 
standard error of the observed trend. DCPS07 do not, which is the main 
error in their analysis.

In calculating modeled and observed standard errors, we assume an AR-1 
model of the regression residuals. This assumption is not unreasonable 
for many meteorological time series. We and others have made it in a 
number of previous studies.

Reviewer 2 would have liked us to fit higher-order autoregressive models 
to the T2, T2LT, and TS-T2LT time series. This is a difficult business, 
particularly given the relatively short length of the time series 
available here. There is no easy way to reliably estimate the parameters 
of higher-order AR models from 20 to 30 years of data. The same applies 
to reliable estimation of the spectral density at frequency zero (since 
we have only 2-3 independent samples for estimating the spectral density 
at frequency zero). Reviewer 2's comments are not particularly relevant 
to the specific problem we are dealing with here.

It's also worth mentioning that use of higher-order AR models for 
estimating trend standard errors would likely lead to SMALLER effective 
sample sizes and LARGER standard errors, thus making it even more 
difficult to find significant differences between modelled and observed 
trends! Our use of an AR-1 model makes it easier for us to obtain 
"DCPS07-like" results, and to find significant differences between 
modelled and observed trends. DCPS cannot claim, therefore, that our 
test somehow stacks the deck in favor of obtaining a non-significance 
trend difference - which they might claim if we used a 
(poorly-constrained) higher-order AR model for estimating standard 
errors.

The Reviewer does not want to "see the method proposed in this paper 
become established as the default method of estimating standard errors 
in climatological time series". We do not claim universal applicability 
of our approach. There may well be circumstances in which it is more 
appropriate to use higher-order AR models in estimating standard errors. 
I'd be happy to make a statement to this effect in the revised paper.

I have to confess that I was a little ticked off by Reviewer 2's 
comments. The bit about "wilfully ignoring" time series literature was 
uncalled for. Together with my former MPI colleague Wolfgang 
Brueggemann, I've fooled around with a lot of different methods of 
estimating standard errors, in both the time domain and frequency 
domain. One could write a whole paper on this subject alone. Such a 
paper would not help us to expose the statistical deficiencies in 
DCPS07. Nor would in-depth exploration of this issue lead to the shorter 
paper requested by the Reviewer.

It should take me a few days to revise the paper and draft a response to 
Reviewer 2's comments. I'll send you the revised paper and draft 
response early next week. Slowly but surely, we are getting there!

With best regards,

Ben
----------------------------------------------------------------------------
Benjamin D. Santer
Program for Climate Model Diagnosis and Intercomparison
Lawrence Livermore National Laboratory
P.O. Box 808, Mail Stop L-103
Livermore, CA 94550, U.S.A.
Tel:   (925) 422-3840
FAX:   (925) 422-7675
email: santer1@llnl.gov
---------------------------------------------------------------------------- 


</x-flowed>
X-Account-Key: account1
Return-Path: <g.mcgregor@auckland.ac.nz>
Received: from mail-1.llnl.gov ([unix socket])
	 by mail-1.llnl.gov (Cyrus v2.2.12) with LMTPA;
	 Tue, 08 Jul 2008 23:07:04 -0700
Received: from mail-1.llnl.gov (localhost.localdomain [127.0.0.1])
	by mail-1.llnl.gov (8.13.1/8.12.3/LLNL evision: 1.7 $) with ESMTP id m69674LM010414
	for <[vacation]santer1@mail.llnl.gov>; Tue, 8 Jul 2008 23:07:04 -0700
Received: (from vacmgr@localhost)
	by mail-1.llnl.gov (8.13.1/8.13.1/Submit) id m69674am010413
	for [vacation]santer1@mail.llnl.gov; Tue, 8 Jul 2008 23:07:04 -0700
X-Authentication-Warning: mail-1.llnl.gov: vacmgr set sender to g.mcgregor@auckland.ac.nz using -f
Received: from nspiron-2.llnl.gov (nspiron-2.llnl.gov [128.115.41.82])
	by mail-1.llnl.gov (8.13.1/8.12.3/LLNL evision: 1.7 $) with ESMTP id m6966u3i010371
	for <santer1@mail.llnl.gov>; Tue, 8 Jul 2008 23:07:03 -0700
X-Attachments: * JOC-production-requ.doc
X-IronPort-AV: E=McAfee;i="5200,2160,5334"; a="21123949"
X-IronPort-AV: E=Sophos;i="4.30,329,1212390000"; 
   d="doc'32?scan'32,208,32";a="21123949"
Received: from nsziron-3.llnl.gov ([128.115.249.83])
  by nspiron-2.llnl.gov with ESMTP; 08 Jul 2008 23:07:01 -0700
X-Attachments: * JOC-production-requ.doc
X-IronPort-AV: E=McAfee;i="5200,2160,5334"; a="47882084"
X-IronPort-AV: E=Sophos;i="4.30,329,1212390000"; 
   d="doc'32?scan'32,208,32";a="47882084"
Received: from uranus.scholarone.com ([170.107.181.135])
  by nsziron-3.llnl.gov with ESMTP; 08 Jul 2008 23:07:00 -0700
Received: from tss1be0007 (tss1be0007 [10.237.148.33])
	by uranus.scholarone.com (Postfix) with ESMTP id B3F62B2883B
	for <santer1@llnl.gov>; Wed,  9 Jul 2008 02:06:59 -0400 (EDT)
Message-ID: <363254525.1215583619734.JavaMail.wladmin@tss1be0007>
Date: Wed, 9 Jul 2008 02:06:59 -0400 (EDT)
From: g.mcgregor@auckland.ac.nz
To: santer1@llnl.gov
Subject: JOC-08-0098.R1 - Decision on Manuscript
Errors-To: g.mcgregor@auckland.ac.nz
Mime-Version: 1.0
Content-Type: multipart/mixed; 
	boundary="----=_Part_1328_363182756.1215583619708"
X-Errors-To: g.mcgregor@auckland.ac.nz
Sender: onbehalfof@scholarone.com

Ben

A couple of comments before you read the standard letter below and associated comments. You have seen those of Referee 1. Now the long awaited comments from reviewer 2 are back. 

The reviewer raises three issues. Of these it would seem to me that length is one issue not to concern yourself with. If you could include a couple of lines clarifying the nature of s(bo) as suggested by the reviewer that would be helpful. Lastly could you address the comment about the time series autocorrelation issue -  I think no further anaysis is required here but what is needed is perhaps a short defence of the strategy adopted.   

I will not send the revised version back to the reviewer but will proceed based on the alterations made to the paper and your associated response summarising how you have addressed the issues raised. Needless to say I look forward to publishing this paper once you have got back to me with the suggested minor alterations completed.
============================
Dear Dr Santer

Manuscript # JOC-08-0098.R1 entitled "Consistency of Modelled and Observed Temperature Trends in the Tropical Troposphere" which you submitted to the International Journal of Climatology, has been reviewed.  The comments of the referee(s), all of whom are leading international experts in this field, are included at the bottom of this letter. If the reviewer submitted comments as an attachment this will only be visible via your Author Centre. It will not be attached to this email. Log in to Manuscript Central, go to your Author Centre, find your manuscript in the "Manuscripts with Decisions" queue. Click on the Decision Letter link. Within the Decision letter is a further link to the reviewer attachment.

I am very happy to grant conditional acceptance of the paper, subject to you making satisfactory revisions as clarified below. These revisions are minor, in the sense that there are no major recalculations or major analyses required, but there may be less-major analyses needed and there are a number of important revisions to the text that are necessary. Please review the attached document listing the file requirements for your revision.

In your revisions, please address all of the points made by each reviewer, make suitable edits to the manuscript, and include a point-by-point response to each reviewer comment with your response.  Use the space provided or upload a file documenting your responses. I will study your responses and determine if the edits were handled satisfactorily. I would like to receive the revised manuscript as soon as is convenient, but with a deadline about 2 months from now.

This should be sufficient time given the minor revisions, but please communicate with me (email is best) now or closer to the time if you need an extension of the deadline.  I feel confident that, pending careful revisions as outlined above, I will be able to publish your paper in the International Journal of Climatology.

You can upload your revised manuscript and submit it through your Author Centre. Log into http://mc.manuscriptcentral.com/joc  and enter your Author Centre, where you will find your manuscript title listed under "Manuscripts with Decisions."

IMPORTANT:  Please make sure you closely follow the instructions for acceptable files. When submitting (uploading) your revised manuscript, please delete the file(s) that you wish to replace and then upload the revised file(s).

Please remember that the publishers will not accept a manuscript unless accompanied by the Copyright Transfer Agreement. Please go to: http://www3.interscience.wiley.com/homepages/4735/nscta.pdf


The Copyright Transfer Form and the Permissions Form should be scanned and uploaded with your submission to Manuscript Central, designated as "Supplemental Material not for review". If you do not have access to a scanner, further instructions will be provided upon the acceptance of your paper. Forms should not be sent to the editorial office.

Once again, thank you for submitting your manuscript to the International Journal of Climatology. I look forward to receiving your revision.

Sincerely,

Prof. Glenn McGregor
Editor, International Journal of Climatology


Referee(s)' Comments to Author:

Referee: 1
Comments to the Author
I think the authors have responded more than satisfactorily to my comments. I have only a few minor additional comments

1. p19, lines 37-39 (short sentence beginning with "This is essentially ..."). I would leave this out I think. DCPS07 do not clearly articulate the hypothesis that they wished to test, so it is slightly misleading to say that they considered a specific hypothesis.

2. The author revised some uses of the word "yields", but seems to have introduced other instances in the conclusions. This is just a matter of style and personal preference, but for me, saying a calculation yields something makes it sound a bit like the author is pleased with the calculation because it produces the anticipated or desired result. 

3. In the acknowledgments, please list my affiliation as "Environment Canada".

Francis Zwiers
21 June 2008
==========================
Referee 2
>From my point of view, there were three major issues about the original
paper -

1. Its length,

2. The failure to distinguish between the two types of test,

3. The treatment of autocorrelation.

On point 1, I still feel the paper is longer than it needs to be, though
as far as I can see, I was the only one of the 2 referees + editor to
raise that as a serious concern. I appreciate the efforts they have made
to remove some tangential material, and it has required some additional
writing to incorporate point 2.

So, if the editor is happy with that, I'm willing to say OK on that point.

On point 2, yes, I'm very satisfied with the work they have done on this.
The two types of hypothesis are now clearly distinguished, and the second
one is laid out in a way that makes clear both the similarities and
differences with the DCPS approach.

It's a little odd that the revised test doesn't seem to make any use of
the individual-model standard errors, equation (4). Instead, what they do
is to compute the sample mean and standard deviation of the estimated
model trends, equations (7) and (9) respectively, and then a separate
estimate of s{bo}, the standard error of the observational trend. I
wondered about this, whether the proposed test would even be correct if
the individual-model standard errors were unequal. In fact, if you square
both sides of equation (9), the sample variance is an unbiased estimator
of the variance of <<bm>> even if the individual variances of the model
estimates are unequal. So the caveat on page 24, lines 6-9, may not be
strictly necessary. I think the test does work (in the form proposed by
the authors, not DCPS), but its potential disadvantage is that maybe
equation (9) is not actually the most efficient way to estimate the
variance of the estimated ensemble mean trend. This might be more of an
issue if the authors were trying to do it with a small number of models
(3, say), rather than the relatively large 19. The test as proposed is
still an order of magnitude improvement on ignoring the observational
variance altogether (the DCPS test), but I don't think it's the last word
on how to do this kind of inter-model comparison, especially when there
are an unequal number of realizations for each model.

By the way, this wasn't what my simulation was about. The only point I was
trying to make there was to show the potential advantage of taking an
ensemble mean viewpoint. That is now clear from the revised paper.

There is one point that I only spotted while reading the revision (though
it was present in the original version) and I think some clarification
from the authors may help. If you read section 4.1, it talks about s{bm}
and s{bo} and then it tells you the formula for s{bm}. But there isn't a
formula for s{bo}. Of course, s{bo} is calculated exactly the same way as
s{bm}, but using the observational rather than the model data. That's
obvious to anyone who understands the basic statistics being used here,
but since the authors' main point is that DCPS didn't use s{bo} at all,
maybe they should take a couple of extra lines to say that explicitly.

Regarding the time series autocorrelation issue, I appreciate the authors'
putting more work into conducting a sensitivity study where they varied
the length of the averaging interval. I have to say, I would have been
more impressed if they had conducted a sensitivity study to different
formulas for calculating the variance of the trend (such as AR(p) for p>1,
or various formulas for approximating the spectral density at frequency
zero, such as the Heidelberger-Welsh procedure which is popular in
operations research). I think what concerns me about the revision is that
there is no mention whatsoever there is a much bigger theory of time
series analysis of which the present treatment is very much a special
case. For example, what if there is long-range dependence? I'm willing to
let go the fact that the proposed method is probably adequate for the
particular time series studied in the present paper, but my concern is
that I wouldn't want to see the method proposed in this paper become
established as the default method of estimating standard errors in
climatological time series. The paper as currently written creates the
impression that the authors are either unaware of the time series
literature or are wilfully ignoring it.

As far as a recommendation on the paper is concerned, I'm willing to
recommend a provisional acceptance. I would urge the authors to give some
further consideration to these points in preparing the final version.


Attachment Converted: "c:\eudora\attach\JOC-production-requ4.doc"

