From: "Michael E. Mann" <mann@virginia.edu>
To: Tom Wigley <wigley@ucar.edu>, Kevin Trenberth <trenbert@cgd.ucar.edu>, Keith Briffa <k.briffa@uea.ac.uk>, Phil Jones <p.jones@uea.ac.uk>, ckfolland@meto.gov.uk, tkarl@ncdc.noaa.gov, jto@u.arizona.edu, mann@virginia.edu
Subject: Fwd: Re: smoothing
Date: Tue, 14 Oct 2003 17:27:24 -0400

   Sorry--one more error. The MSE values for "minimum norm" and "minimum roughness" are
   switched in the figure legend. Obviously the former is a better fit...
   mike

     Date: Tue, 14 Oct 2003 17:08:49 -0400
     To: Tom Wigley <wigley@ucar.edu>, Kevin Trenberth <trenbert@cgd.ucar.edu>, Keith Briffa
     <k.briffa@uea.ac.uk>, Phil Jones <p.jones@uea.ac.uk>, ckfolland@meto.gov.uk,
     tkarl@ncdc.noaa.gov, jto@u.arizona.edu, mann@virginia.edu
     From: "Michael E. Mann" <mann@virginia.edu>
     Subject: Re: smoothing
     Bcc: Scott Rutherford <srutherford@rwu.edu>
     correction '1)' should read:
     '1) minimum norm: sets padded values equal to mean of available data beyond the
     available data (often the default constraint in smoothing routines)'
     sorry for the confusion,
     mike
     At 05:05 PM 10/14/2003 -0400, Michael E. Mann wrote:

     Dear All,
     To those I thought might be interested, I've provided an example for discussion of
     smoothing conventions.  Its based on a simple matlab script which I've written (and
     attached) that uses any one of 3 possible boundary constraints [minimum norm, minimum
     slope, and minimum roughness] on the 'late' end of a time series (it uses the default
     'minimum norm' constraint on the 'early' end of the series). Warming: you needs some
     matlab toolboxes for this to run...
     The routines uses a simple butterworth lowpass filter, and applies the 3 lowest order
     constraints in the following way:
     1) minimum norm: sets mean equal to zero beyond the available data (often the default
     constraint in smoothing routines)
     2) minimum slope: reflects the data in x (but not y) after the last available data
     point. This tends to impose a local minimum or maximum at the edge of the data.
     3) minimum roughness: reflects the data in both x and y (the latter w.r.t. to the y
     value of the last available data point) after the last available data point. This tends
     to impose a point of inflection at the edge of the data---this is most likely to
     preserve a trend late in the series and is mathematically similar, though not identical,
     to the more ad hoc approach of padding the series with a continuation of the trend over
     the past 1/2 filter width.
     The routine returns the mean square error of the smooth with respect to the raw data. It
     is reasonable to argue that the minimum mse solution is the preferable one.  In the
     particular example I have chosen (attached), a 40 year lowpass filtering of the CRU NH
     annual mean series 1856-2003, the preference is indicated for the "minimum roughness"
     solution as indicated in the plot (though the minimum slope solution is a close 2nd)...
     By the way, you may notice that the smooth is effected beyond a single filter width of
     the boundary. That's because of spectral leakage, which is unavoidable (though minimized
     by e.g. multiple-taper methods).
     I'm hoping this provides some food for thought/discussion, esp. for purposes of IPCC...
     mike
     ______________________________________________________________
                         Professor Michael E. Mann
                Department of Environmental Sciences, Clark Hall
                           University of Virginia
                          Charlottesville, VA 22903
     _______________________________________________________________________
     e-mail: mann@virginia.edu   Phone: (434) 924-7770   FAX: (434) 982-2137
              [1]http://www.evsc.virginia.edu/faculty/people/mann.shtml

     ______________________________________________________________
                         Professor Michael E. Mann
                Department of Environmental Sciences, Clark Hall
                           University of Virginia
                          Charlottesville, VA 22903
     _______________________________________________________________________
     e-mail: mann@virginia.edu   Phone: (434) 924-7770   FAX: (434) 982-2137
              [2]http://www.evsc.virginia.edu/faculty/people/mann.shtml

   ______________________________________________________________
                       Professor Michael E. Mann
              Department of Environmental Sciences, Clark Hall
                         University of Virginia
                        Charlottesville, VA 22903
   _______________________________________________________________________
   e-mail: mann@virginia.edu   Phone: (434) 924-7770   FAX: (434) 982-2137
            [3]http://www.evsc.virginia.edu/faculty/people/mann.shtml

References

   1. http://www.evsc.virginia.edu/faculty/people/mann.shtml
   2. http://www.evsc.virginia.edu/faculty/people/mann.shtml
   3. http://www.evsc.virginia.edu/faculty/people/mann.shtml

