date: Mon Apr 25 11:19:18 2005
from: Phil Jones <p.jones@uea.ac.uk>
subject: Re: Report
to: "Brohan, Philip" <philip.brohan@metoffice.gov.uk>, Ian Harris <i.harris@uea.ac.uk>

    Philip,
       Harry and I are meeting at 2pm today and you'll get the answers and/or
    additional material later today or tomorrow.  As I'm away tomorrow, I'll
    answer 3 now and give my thoughts on 6.
    3. Each month the HC updates the 1991-2005 file with the latest CLIMAT messages.
    I also do this and pick up earlier CLIMAT messages (for the last 18 months) or
    so in case any late data get added in. I do this monthly and add in any months
    of MCDW (Monthly Climatic Data for the World - a file I get from NCDC) that
    have been received in the past month. MCDW is currently about 2-3 months behind
    real time. About 100 stations get added through this - most are additional stations
    in the US that don't appear in the GTS CLIMAT messages.
     I also from MCDW add in their back data. This is approximately an additional 50-100
    station months worth of data - for a random selection of countries. This often (but not
    always) includes Pakistan, a few sites in India, Sudan, Brazil, a few sites in
    Thailand plus others. These listed have been regulars in the last 12 months.
     How NCDC gets this data isn't clear to me. It is likely to be back data received after
   the
    cut-off each month. It could be mailed in reports. If it is coming by the GTS your system
    could be losing it.  In assessments done by WMO every year, there is a +/- 2-5% difference
    in CLIMAT reports received at the main monitoring centres around the world.
      Each month I also manually add in monthly temperatures for some sites in the Antarctic.
    These are for the South Pole and Russian sites on the continent (3 sites, Bellingshausen,
    Vostok and Mirny). South Pole never comes in over the GTS and the Russian sites are
    very sporadic. I get these by email from the SP and by email from a colleague in St.
    Petersburg (Victor Lagun).  Victor is on the attached paper which came out in IJC
   recently.
    All the Antarctic data in this paper are in the dataset.  Every now and then I go to the
   BAS
    web site of Gareth Marshall and pick up and manually add in back Antarctic data (this is
    mainly for the French site at Dumont D'Urville)  and the NZ site (Scott Base) as reporting
   from
    other countries has been relatively good recently.
       I don't keep any sort of record of any of this, but all the manually entered data is
   checked
    by me for outliers. All the MCDW data and back CLIMATs entered automatically get their
    anomalies listed out and I look for largish values.
      I am happy to continue doing all this after HadCRUT3 starts.  The Antarctic stuff is
   important
    as coverage is poor down there. NCDC have produced maps of recent trends for the AR4 from
    their dataset and their Antarctic coverage is much poorer than HadCRUT2. We are currently
    trying to figure out if this is because of missing data or the their plotting software. It
   may be
    that the single 5 by 5 boxes get omitted when plotting.
    6)  What Harry has plotted is the annual average corrections in the histograms. The CRU
    averages appear small, but the stations are all over the world, so averaging makes them
    appear near zero - adjustments don't have a preferential sign (which might be a point
   worth
    making, if you gridded with and without the adjustments, differences would be very small).

     For Canada, although a big country, it is still only a small part of the world. The
   adjustments
    they have applied tend to make sites warmer or colder by about 0.7 deg C. There is less
    cancelling going on and the adjustments fall into two camps.
      Maybe if we split the CRU adjustments into continents (even just NH and SH) we might
    get the bimodal shapes that Canada shows.  The different seasonal cycles around the world
    partly produce the CRU result as for winter months adjustments tend to be larger.  I'll
   see
    if Harry can try this for USA+Central America/Caribbean,  S. America, Europe, Asia
    Australasia and Africa (+NH and SH).  Some of these may not have many samples though.
    Cheers
    Phil
   At 10:07 25/04/2005, Brohan, Philip wrote:

     Hi Harry.
      Thanks for the report, it is basically fine - clear and concise. It
     describes well all the good work you've done on the data.
      There are a few points that still need clearing up, mostly details
     needed for the HadCRUT3 paper I am writing:
      1) I am using revision 10 of the data files (data, normals and SDs). Is
     this the final version? If not, please send me the latest revision.
      2) Have you any references for the data changes you have made? We'll
     need these for the paper.
      3) Have any other station data changes or additions been made since the
     production of HadCRUT2? (Antarctic data?) This isn't really part of your
     report, but we'll need to put details in the paper.
      4) Please send me a list of the station IDs you deleted. I can then
     delete them from the recent data file as well.
      5) You've done a good job cleaning up the normals, but I still need
     estimates of the expected uncertainty of the station normals where they
     are inferred from limited data or taken from WMO. You will know this
     from the work you describe in section 3 on process improvement. Can you
     add a couple of figures like those in section 4, showing the difference
     between the WMO and new normals for your 617 stations, and the effect of
     using only 15 of the 30 years in generating a climatology?
      6 I'm a bit confused by section 4: Additional information. If I
     interpret it correctly, for both the CRU and Canadian data, a bit under
     half of the measurements have been corrected. But the CRU corrections
     are most likely to be small (~.2C), while the Canadian corrections are
     never small, and are most likely to be medium sized (~.6C). Is this
     right? Can you shed any light on this discrepancy?
      (At first glance it looks as if the uncorrected Canadian data has large
     inhomogeneities which are approximately normally distributed, but only
     those larger than a threshold have been identified and corrected; while
     the CRU corrections are an attempt to correct for the remaining small
     inhomogeneities. But this is pure speculation on my part.)
     Thanks,
      Philip
     On Fri, 2005-04-22 at 18:56, Ian Harris wrote:
     > Philip,
     >
     > Please find attached a final draft of the report, "ENHANCEMENT AND
     > QUALITY CONTROL OF CRU MONTHLY TEMPERATURE STATION DATA SET'.
     >
     > All comments, etc will be dealt with as quickly as possible.
     >
     > Cheers
     >
     > Harry
     > Ian "Harry" Harris
     > Climatic Research Unit
     > University of East Anglia
     > Norwich NR4 7TJ
     > United Kingdom
     --
     Philip Brohan,  Climate Scientist
     Met Office   Hadley Centre for Climate Prediction and Research
     Tel: +44 (0)1392 884574    Fax: +44 (0)1392 885681
     Global climate data sets are available from [1]http://www.hadobs.org

   Prof. Phil Jones
   Climatic Research Unit        Telephone +44 (0) 1603 592090
   School of Environmental Sciences    Fax +44 (0) 1603 507784
   University of East Anglia
   Norwich                          Email    p.jones@uea.ac.uk
   NR4 7TJ
   UK
   ----------------------------------------------------------------------------

