cc: Caspar Ammann <ammann@ucar.edu>, rbradley@geo.umass.edu, tcrowley@duke.edu, mhughes@ltrr.arizona.edu, omichael@princeton.edu, t.osborn@uea.ac.uk, jto@u.arizona.edu, Scott Rutherford <srutherford@rwu.edu>, Tom Wigley <wigley@ucar.edu>, p.jones@uea.ac.uk, ckfolland@meto.gov.uk, tkarl@ncdc.noaa.gov
date: Mon, 13 Oct 2003 14:21:14 -0400
from: "Michael E. Mann" <mann@virginia.edu>
subject: Re: draft
to: Keith Briffa <k.briffa@uea.ac.uk>, Kevin Trenberth <trenbert@cgd.ucar.edu>

   Dear Keith,
   Thanks a bunch for your comments, all of which are very helpful. I've attempted to
   incorporate these, along w/ those of Tim, Tom W, in the latest draft, as per my previous
   email.
   There is one point I wanted to comment on further, regarding the issue the how a
   potentially non-stationary time series (e.g. one with a significant trend near the end)
   should or should not be smoothed. The issue, happily, is not relevant to our Eos reply,
   because the proxy reconstruction (which ends in 1980) was smoothed based on a procedure
   that does not assume a continuation of the trend, the issue that seems controversial here.
   However, I do think that this is particularly important in smoothing of the instrumental
   surface temperature series. Those uninterested in this particular discussion need not read
   any further, but I would encourage those interested (particular as it might involve
   decisions about how to smooth time series in the next IPCC report) to read on...
   mike
   comment on "minimum roughness" constraint in smoothing time series with significant trends
   near the end of the series:
   I favor a "minimum roughness" constraint (which tends to retain trends near the end of a
   smooth)  for smoothing of  a series with a significant trend near (either) end for the
   following reason. Smoothing of a time series with a zero phase (i.e., centered) filter
   reflects a non-unique transformation of the data. It is non-unique because there is no
   information on one half of the filter center at either the beginning or end of the series.
   Because of that lack of information, an additional a priori constraint has to be placed on
   the filtering process. This contraint reflects an assumption about the data outside of the
   available interval. This can be cast (as often it is in the signal processing literature)
   as an inverse problem w/ non--unique constraints. The typical constraints that are
   typically employed [see Park, J., Envelope estimation for quasi-periodic geophysical
   signals in noise: A multitaper approach, in Statistics in the Environmental and Earth
   Sciences, edited by A. T. Walden and P. Guttorp, pp. 189-219, Edward Arnold, London, 1992.]
   involve a (i) "minimum norm", (ii) "minimum slope", and (iii) "minimum roughness" solution
   for the underlying statistical model. A possible way of insuring a reasonably objective
   smoothing of the time series is to minimize, among all possible linear combinations of
   these 3 models, that which minimizes the mean-square-misfit with respect to the raw data
   [see  Ghil, M., Allen, M.R., Dettinger, M.D., Ide, K., Kondrashov, D., Mann, M.E.,
   Robertson, A.W., Tian, Y., Varadi, F., Yiou, P., Advanced Spectral Methods for Climatic
   Time Series, Reviews of  Geophysics, 40 (1), 1003, doi: 10.1029/2000RG000092, 2002.]
   Such an approach will favor contraints (i) or (ii) for a data series with stationary
   behavior in the mean near the end, and probably (iii) if there is non-stationary behavior
   (i.e. a long-term trend or, more specifically, a statistically signfiicant trend over the
   last 1/2 smoothing filter window width).
   (i) is called the "minimum norm" constraint because it chooses the smallest of all models
   for the smoothed data--it involves the minimization of the 0th derivative of the smooth.
   The implicit assumption is that the mean outside the available interval is equal to the
   mean of the available data. This is clearly wrong if there is a signfiicant trend.
   (ii) is called a "minimum slope" inversion. In this case, the solution involves the
   minimization of the mean-square first derivative of the model--the solution will favor a
   smooth that approaches the boundary with zero slope--the implicit assumption is that the
   mean outside the available interval may be different from the mean inside the available
   data, but that this reflects a step change in the mean value rather than any trend near the
   boundary. Ad hoc methods which pad the end of the series with e.g. the mean of the last 1/2
   filter width, in essence, implement this constraint.
   (iii) is called the "minimum roughness" solution because it minimizes the mean-square 2nd
   derivative among all possible models. It favors a smooth with an inflection point at the
   boundary. and is consistent with the assumption that a trend exists as one approaches the
   boundary. Mathematically, it is most simply implemented in the time domain by padding the
   series with an extension of the trend over the past 1/2 filter width. However, the
   constraint can be implemented directly in the frequency-domain inversion [Park, 1992; Ghil
   et al, 2002].
   This is the proper choice if there is a statistically significant trend within the final
   1/2 smoothing window width of the edge of the data. Objectively, it is defensible in those
   situations where this choice minimizes the mean-square misfit with the raw data over all
   possible linear combinations of choices (i), (ii), and (iii). For the global or hemispheric
   mean instrumental series from 1856-present, that condition holds.
   So my concern is the opposite of Keith's. I believe that smoothing routines that explicitly
   invoke an assumption of stationarity are problematic when the series clearly is not
   stationary.
   This assertion has a rigorous foundation in the inverse theory literature [see Park, 1992
   and other references therein] and is not a new or subjective approach.
   Though the point is actually irrelevant to the discussion at hand for reasons mentioned
   earlier, I would actually like to see this discussed, because I think that we (e.g. in IPCC
   '01) may have underplayed the significance of recent warming by employing improper boundary
   conditions on smooths of records like the global temperature or hemispheric temperature
   series. I've cc'd Chris Folland and Tom Karl in on this discussion, for their comments...

     Mike and all
     Hi , just back from a trip and only now catching up with important emails. Given
     the restricted time and space available to furnish a response to SB comments ,
     I offer the following mix of comment and specific wording changes:
     I agree that the S+B response is designed to deflect criticism by confusing the issues
     rather than answering our points.
     In fact they fail to address any of the 3 specific
     issues we raised Namely , 1. the need for critical evaluation of proxy inputs , 2. the
     need for a consistent assimilation of widespread (dated and well resolved ) records,
     3. the essential requirement for objective/quantitative calibration (scaling) of the
     input
     records to allow for assessment of the uncertainties when making
     comparisons of different reconstructions and when comparing early with recent
     temperatures.
      Their own , ill-conceived and largely subjective approach did not take
     account of the uncertainties and problems in the use of palaeodata that they chose to
     highlight in their opening remarks.
     I would be in favour of stating something to this effect at the outset of our response.
     Also , as regards the tree-ring bit , I fully concur with  the sense of your text as
     regards Section 1, but suggest the following wording (to replace ",rarely for annual
     ring widths, and almost entirely at higher latitudes.")
     "but in certain high-latitude regions only. Where this is the case , these relatively
     recent
     (ie post 1950) data are not used in calibrating temperature reconstructions. In many
     other
     (even high-latitude) areas  density or ring-width records display no bias."
     In the spirit of healthy debate - I agree with Tim's remarks , warning against
     presenting a too
     sanguine impression that the borehole debate is closed ( though I do think it is
     closing!).
     I also believe , as you already know, that the use of a recent padding algorithm to
     extend
     smoothed data to the present time, is inappropriate if it assumes the continuation of a
     recent
     trend. This is likely to confuse , rather than inform, the wider public about the
     current climate state .
     Finally , I repeat my earlier remarks (made before EOS piece published) that we are
     missing
     an opportunity to say that a warm Medieval period per se is not a refutation of
     anthropogenic
     warming , {as its absence is no proof}, if we do not understand the role of specific
     forcings (natural
     and anthropogenic) that influenced medieval and current climates.
     Cheers
     Keith
     At 12:48 PM 10/9/03 -0600, Kevin Trenberth wrote:

     Hi all
     Here are my suggested changes: toned down in several places.  Tracking turned on
     Kevin
     Michael E. Mann wrote:

     Dear co-authors,
     Attached is a draft response, incorporating suggestions Kevin, Tom W, and Michael.  I've
     aimed to be as brief as possible, but hard to go much lower than 750 words and still
     address all the key issues. 750 words, by the way, is our allotted limit.
     Looking forward to any comments. Feel free to send an edited version if you prefer, and
     I'll try to assimilate all of the suggested edits and suggestions into a single revised
     draft. If you can get comments to me within the next couple days, that would be very
     helpful as we're working on a late October deadline for the final version.
     Thanks for your continued help,
     mike
     ______________________________________________________________
                         Professor Michael E. Mann
                Department of Environmental Sciences, Clark Hall
                           University of Virginia
                          Charlottesville, VA 22903
     _______________________________________________________________________
     e-mail: <[1]mailto:mann@virginia.edu >mann@virginia.edu   Phone: (434) 924-7770   FAX:
     (434) 982-2137
              [2]http://www.evsc.virginia.edu/faculty/people/mann.shtml

     --
     ****************
     Kevin E. Trenberth                              e-mail:
     <[3]mailto:trenbert@ucar.edu>trenbert@ucar.edu
     Climate Analysis Section, NCAR
     <[4]http://www.cgd.ucar.edu/cas/>[5]www.cgd.ucar.edu/cas/
     P. O. Box 3000,                                 (303) 497 1318
     Boulder, CO 80307                               (303) 497 1333 (fax)
     Street address: 1850 Table Mesa Drive, Boulder, CO  80303

     --
     Professor Keith Briffa,
     Climatic Research Unit
     University of East Anglia
     Norwich, NR4 7TJ, U.K.
     Phone: +44-1603-593909
     Fax: +44-1603-507784
     [6]http://www.cru.uea.ac.uk/cru/people/briffa/

   ______________________________________________________________
                       Professor Michael E. Mann
              Department of Environmental Sciences, Clark Hall
                         University of Virginia
                        Charlottesville, VA 22903
   _______________________________________________________________________
   e-mail: mann@virginia.edu   Phone: (434) 924-7770   FAX: (434) 982-2137
            [7]http://www.evsc.virginia.edu/faculty/people/mann.shtml

