cc: Kate Willett <kate.willett@metoffice.gov.uk>, Dick Dee <Dick.Dee@ecmwf.int>
date: Mon Dec 22 15:54:06 2008
from: Phil Jones <p.jones@uea.ac.uk>
subject: Re: CRUTEM3v and HadCRUH comparisons with ERA-40 and ERA-Interim
to: Adrian.Simmons@ecmwf.int, "peter.thorne" <peter.thorne@metoffice.gov.uk>

    Adrian,
       The grid box count plots you've sent clearly indicate to me that
    we will need to do something to CRUTEM3v. I knew we had a problem
    going from CRUTEM3 to CRUTEM3v, which affected the Antarctic. This
    influences SH extratropics as well. A couple of the other regions
    have seasonal cycles in the counts, which they shouldn't.
        John Kennedy at MOHC now has a corrected file (from CRU) that
    won't have this problem, but I don't know when it will be implemented.
    It is a long story as to why it happened.
       A simple way for you to get over it is to use CRUTEM3 - with no v.
       All this won't make any difference to your earlier plots except for the Antarctic,
    and then only possibly.
      I'll get on to John in the New Year to rerun CRUTEM3v.
    Cheers
    Phil
   At 12:25 22/12/2008, Adrian Simmons wrote:

     Thanks for the comments, Peter. A few quick responses:
     (i) I'll check different tolerances for including CRUTEM3v data, but
     doubt it will help coverage much. Actual numbers can be seen in the
     attached plots. They shift mostly only on the decadal timescale,
     consistent with the 10-year book explanation given by Phil earlier.
     (ii) For temperature, I could show the CRUTEM2v time series plus
     differences. To see anything much though I would have to use a different
     scale for differences, and that may detract from the main point, which
     is that the time series are very similar indeed. I would argue that if
     you can't see the differences, they can't be very significant. I've got
     a couple of tables to quantify differences. I'll try out the alternative
     nevertheless. This approach won't work for humidity, however, as none of
     the time series spans the full period.
     (iii) If you could extend HadCRUH over land to the end of 2007, albeit
     in provisional form, I would certainly be interested.
     (iv) I wondered myself about flipping the colour for the humidity plots.
     But q increases as T increases. So I thought to use red colours both for
     larger T (conventional) and larger q (unconventional, I know), so that
     the correlation stands out. I guess it's a matter of taste. If we stick
     with the current scheme, I would include a sentence explaining why blue
     means dry.
     (v) I had a longish chat with June Wang when I visited NCAR in October.
     She was then about to put in a proposal jointly with Aiguo Dai (who I've
     not met) and Leo for funding for an attempt at homogenising radiosonde q
     - something we're very interested in for future reanalysis. I've also
     just looked at a paper of hers on the dry RS80 bias, as this may play a
     part in the RH shift over Europe either side of ~1990 (generally,
     radiosondes dry our boundary-layer analysis whilst SYNOPs moisten it). A
     visit from them in the summer would be very welcome.
     Best regards
     Adrian
     peter.thorne wrote:
     > Dear Adrian,
     >
     > some comments below inter-leaved within the text. Hopefully they will be
     > useful.
     >
     >
     > On Fri, 2008-12-19 at 18:38 +0000, Adrian Simmons wrote:
     >> Dear Phil, Kate and Peter
     >>
     >> I finally managed to spend some time over the past few days assembling
     >> some provisional figures (attached) for a comparison of our reanalyses
     >> with your data sets. Era-Interim is now in January 2008, and when it
     >> completes 2008 (probably late next month or early February), we'll have
     >> 20 years of this reanalysis, and thirty years of combined
     >> ERA-40/ERA-Interim reanalysis from the FGGE year onwards. My idea is to
     >> start writing something soon, with a view to finalising the figures and
     >> text once we have results up to the end of 2008.
     >
     > Sounds great.
     >
     >> Here are some comments on the figures (all of which are land-only).
     >>
     >> (i) Figs. 1-3 These cover the temperature comparisons, and simply update
     >> the picture from Simmons, Jones et al. (2004). In these and other
     >> figures I have normalised everything to give zero anomaly for the
     >> ten-year mean 1989-1998, which is convenient as I can then plot maps of
     >> the anomalies for the decades 1979-1988 and 1999-2008. (Note that for
     >> now the maps labelled 1999-2008 are actually for 1998-2007, and those
     >> for 2004-2008 are actually for 2003-2007 - this will be fixed when we
     >> actually have reanalysis data for 2008). For the maps I show all CRU
     >> grid squares for which less than 5% of months are missing.
     >>
     >
     > Is there any reason that the criteria is this tight? Is there any
     > implication to loosening it to say 25%? You should find extra data in at
     > least some relatively data sparse areas which may be worth the cost of
     > the additional noise, or does it make no substantial difference to
     > coverage for either dataset?
     >
     >> Basically we see excellent continuation of the ERA-40 results when we
     >> sample ERA-Interim and ERA-40 in the same way as CRUTEM3v samples in
     >> space and time. A point that might be emphasized is that the coverage of
     >> CRUTEM3v is much poorer for the last ten years than the first one. Any
     >> comment on this? Are there many late-arriving CLIMATs that will change
     >> the picture when I download a new version of CRUTEM3v? It is noteworthy
     >> that CRUTEM3v samples poorly the Arctic region where ERA indicates the
     >> strongest warming (and ERA also shows strong warming over sea-ice - or
     >> where sea-ice used to be!). Fig 1 shows a stronger trend in ERA that in
     >> CRUTEM3v when we use the complete ERA record rather than sampling it as
     >> CRUTEM3v. In other words ERA-Interim shows less recent cooling than
     >> CRUTEM3v. Here we are treading on thin ice (sorry!) from a PR viewpoint,
     >> so I'm interested in your reaction to all this.
     >
     > For all the timeseries figures or at least those that you subsample to
     > coverage mimicking can we consider a slightly different presentational
     > style? I find it hard to ascertain what is going on because the lines
     > are sufficiently thick to overlap substantially for much of the period.
     > An alternative approach would be to show one dataset's (the
     > "observations"? Open to argument and counter-vailing views here) actual
     > timeseries and then offset from this the difference series to the other
     > two timeseries. We know the datasets agree on the high frequency, its
     > the low frequency behaviour that really interests us here. The
     > difference series illuminates this directly and could be assessed for
     > statistical trend significance probably without a d.o.f. correction to
     > the uncertainty. This would very considerably strengthen this part of
     > the analysis as it is actually really getting at a very thorough
     > investigation of the inter-dataset differences which is key.
     >
     >> (ii) Fig. 4 shows excellent agreement between HadCRUH and ERA-40 time
     >> series for q. After 2003, the "sampled as HadCRUH" means I sample
     >> ERA-Interim for the grid squares for which HadCRUH gives values for
     >> December 2003. Slightly less trend for the full ERA dataset.
     >>
     >> (iii) Fig. 5 and 6 are time series for RH. Generally good agreement
     >> also. Two points. One is shift in ERA-40 around 1990 for Europe. I think
     >> this is a reanalysis problem. Hard to be sure, but could be explained by
     >> increased dry bias of sondes. No time to write more now, but Figs 10 and
     >> 11 relate to this, and also to second point - the relative dryness of
     >> recent years. I cannot find a reanalysis problem to explain this, and am
     >> inclined to think for now it could be real - there is no fundamental
     >> physical reason why relative humidity should not show a trend in a
     >> climate that is shifting. But it is really frustrating that HadCRUH
     >> stops at the end of 2003. What are the prospects for extending it? Could
     >> something quick be done as a check on the reanalysis result - omitting
     >> homogenization and subtle QC checks for example.
     >
     >
     > We could, in theory, update the series as we now have, just,
     > unquarantined disk with the ISD source updated through the end of 2007
     > (wouldn't help with '08). We could, as an interim measure, update
     > through 2007 by simply plonking it on the end for now (not quite that
     > simple as we'd have to convert to required res and format but the hard
     > yards are done). Adding 2008 would be technically feasible but a heck of
     > a lot of work and is probably outside the time remit. We were going to
     > produce a 2nd gen product for end of '09 through 2007 then worry about
     > instigating updates. That's quite a lot to do, but I guess in theory we
     > have the data through 2007 so could update the land component thru then.
     > Not sure about marine, but that's not your problem ...!
     >
     >> (iv) Fig 7 shows ten-year anomaly maps. Don't have HadCRUH for 1999-2008
     >> of course. RH map for 1999-2008 show good temporal continuity -
     >> generally drying (relatively) in tropics and mid latitudes, and
     >> moistening (relatively) at high latitudes. Perhaps not implausible - for
     >> RH to stay uniform  as T increases the water has to come from somewhere.
     >> That could be difficult over dry soils. And easier over melting soils??
     >> Is anything like this seen in the Hadley Centre climate runs?
     >>
     >
     > In the humidity maps can you flip the colour scheme so that red is drier
     > and blue is wetter? That is more intuitive to me at least.
     >
     > We haven't had an in depth look at climate model output in this respect.
     > We should have some HadGEM1 fields that we could dig out, but Kate's
     > been doing some HadCRUH-climate model work and we want to ensure we
     > don't duplicate so suggest Kate updates you on status.
     >
     >> (v) Fig 8 shows consistency across boundary layer (level 49 is close to
     >> 850hPa, where analysis in influenced more strongly by sondes rather than
     >> SYNOPs).
     >>
     >> (vi) Fig 9 shows some 5-year means, comparing ERA-INT and HadCRUH for
     >> 1999-2003.
     >>
     >> (vii) Figs 10 and 11 are fits of background and analysis to assimilated
     >> observations. There are subtleties I don't have time to explain right
     >> now, but basically a flat line is a good thing from the viewpoint of
     >> trend analysis. It is the drop in the background RH curves and rise in
     >> the background Q curves for the northern hemisphere between 1988 and
     >> 1993 that makes me think the RH shift between the 1980s and 1990s in ERA
     >> relative to HadCRUH is a reanalysis problem. Equally though, there is
     >> nothing in these curves (or some others I've looked at) that points to
     >> the lowering of RH in the reanalyses for the last few years as being
     >> spurious.
     >
     > Which makes it more appealing to do a quick and dirty fix to update
     > HadCRUH through at least 2007 as that would be the clincher. Of course,
     > in this era we also have the GPS-PW network from June Wang at NCAR.
     > Perhaps this would be a useful addition? Aiguo Dai also has an
     > alternative observational dataset of surface q and RH which could be a
     > useful addition. I think his is updated through 2006. It has no
     > homogeneity adjustments but because its a compilation of synops probably
     > better coverage (swings and roundabouts). We have both of them visiting
     > here in the summer for 2 months and 3 months respectively so opportunity
     > to do some collaborative work on this. I know they were keen to visit
     > CRU and Reading institutes. If you want their email addresses so you can
     > send some enquiries just let me know.
     >
     > I hope you have a relaxing christmas
     >
     > Peter
     --
     --------------------------------------------------
     Adrian Simmons
     European Centre for Medium-Range Weather Forecasts
     Shinfield Park, Reading, RG2 9AX, UK
     Phone: +44 118 949 9700
     Fax:   +44 118 986 9450
     --------------------------------------------------

   Prof. Phil Jones
   Climatic Research Unit        Telephone +44 (0) 1603 592090
   School of Environmental Sciences    Fax +44 (0) 1603 507784
   University of East Anglia
   Norwich                          Email    p.jones@uea.ac.uk
   NR4 7TJ
   UK
   ----------------------------------------------------------------------------
