cc: p.jones@uea.ac.uk, i.harris@uea.ac.uk, t.osborn@uea.ac.uk
date: Tue, 22 Apr 2008 12:54:53 +0100 (BST)
from: P.Jones@uea.ac.uk
subject: Re: [Fwd: Re: {Spam?} Re: Status]
to: "David Lister" <D.Lister@uea.ac.uk>


 David,
   Looks as though the time series data are wrong then
 for 61-96 assuming that you've not updated the earlier data
 with VAP data, where the earlier was something other than VAP.

  As I said earlier the time series data for the Gabon sites
 for VAP look like temperature data from somewhere like Ireland.

   Hope you can sort this out!

 Cheers
 Phil


> Phil/Harry/Tim,
>
> I have looked in Mark New's normals database (I have not yet looked in the
> "red book").  The Gabon stations are listed below:
>
> 6450000GAB8795 1719  301  308  302  309  304  274  248  250  268  288  297
> 306
> 6450300GAB6190100-1  287  289  296  308  294  242  223  232  254  277  287
> 277
> 6450400GAB6190100-1  298  301  305  301  301  279  255  261  286  287  294
> 289
> 6451000GAB8795 1419  272  256  255  260  263  256  247  243  249  260  261
> 258
> 6454500GAB6190100-1  239  241  247  250  249  242  226  226  240  244  243
> 257
> 6455000GAB8795 1119  301  299  294  301  305  274  249  250  267  283  290
> 301
> 6455200GAB6190100-1  261  264  266  253  268  254  237  236  249  254  258
> 249
> 6455600GAB6190100-1  251  249  253  255  250  239  227  226  239  244  243
> 240
> 6456500GAB6190100-1  227  263  268  268  260  230  219  226  248  261  263
> 258
>
>
> These values look reasonable to me.  I will check further if required.
> NB four of the above (code="-1") are derived variables.  The others come
> from CRU's GPD.
>
> Cheers
>
> David
> On Tue, 22 Apr 2008 P.Jones@uea.ac.uk wrote:
>
>> ---------------------------- Original Message
>> ----------------------------
>> Subject: Re: {Spam?} Re: Status
>> From:    f028@uea.ac.uk
>> Date:    Tue, April 22, 2008 7:45 am
>> To:      "Ian Harris" <i.harris@uea.ac.uk>
>> Cc:      "Tim Osborn" <t.osborn@uea.ac.uk>
>>         "Phil Jones" <p.jones@uea.ac.uk>
>> --------------------------------------------------------------------------
>>
>>
>> Tim, Harry,
>>   First can't see why Harry's email has come as spam.
>>
>> A few thoughts on the data for the 2 Gabon stations.
>> If these are supposed to VAP data then the data for the
>> period 1961-1996 looks completely wrong. The later data looks
>> better!
>>
>>  Seems as though these two sites and the others for the country
>> ought to be removed. I'm making this suggestion based on
>> common sense. Gabon is near the equator and I think most is
>> jungle.
>>
>>   I could be wrong. The Sudan VAP for El Dueim looks OK. It should
>> drop during teh dry season.
>>
>>   Maybe ask David to have a look. Or have a look at a station in an
>> adjacent country.
>>
>>  No idea what the Gabon numbers are for the 61-96 period! They look
>> like temperatures from somewhere like Ireland!
>>
>>   Hope you can find the problem!
>>
>> Cheers
>> Phil
>>
>>
>>> Hi Tim,
>>>
>>> A little more detail now I've had a chance to look at the files. Try
>>> and view this email in a monospaced font! Or it'll look awful.
>>>
>>> On 21 Apr 2008, at 16:23, Tim Osborn wrote:
>>>
>>>> Hi Harry,
>>>>
>>>> yes, please focus on WET first.
>>>>
>>>> then please take a look at the VAP station data in the countries of
>>>> Gabon (1st priority) and Sudan (2nd priority), for 2003 onwards.
>>>>
>>>> I've checked the synthetic vap and that does not have extremely high
>>>> recent values, so it seems to be coming from the stations.  I masked
>>>> out areas not near to stations and the high extremes are still
>>>> there... so that's further evidence it is in the stations.  Gabon
>>>> shows particularly strong anomalies (approaching 20 hPa above normal
>>>> at times), but there are some over 10 hPa above normal in the Sudan
>>>> area too.
>>>>
>>>> The two ncview.*.pdf files attached show the anomaly maps for 2
>>>> selected months during 2003-2005 (see file names for specific
>>>> months/years).  I've masked in dark blue areas outside the CDD
>>>> distance from a station (according to your ...vap.stn.nc file).  The
>>>> high values are red (surrounded a yellow stripe).  Gabon in west
>>>> equatorial Africa stands out in both, Sudan in one.  Both have
>>>> numerous other high anomalies like this from Jan 2003 onwards.  They
>>>> are the cause of spikes in country-mean VAP in the other attached file
>>>> -- see page 15 for Gabon (and neighbouring countries) and page 13 for
>>>> Sudan (and neighbouring countries).  Sudan looks less severe, but
>>>> that's partly because it is larger and the averaging takes in more
>>>> synthetic vap values from outside the station's neighbourhood.
>>>>
>>>> The extreme positive anomalies occur close to (this is from visual
>>>> reading off the screen, so not fully precise!):
>>>>
>>>> Gabon: 9.25 E, 0.25 S -- is there a station at Port-Gentil?
>>>
>>> There is - well, three, as this is a database I haven't cleansed:
>>>
>>> -645014  -187   1102   89 MOUILA               GABON         1971 1996
>>>   -999  -999.00
>>> -645009    50    940   15 LIBREVILLE           GABON         1971 1996
>>>   -999     0.85
>>> -645008   -70    875    4 PORT GENTIL          GABON         1971 2002
>>>   -999     -999
>>> -645007   -70    875    4 PORT GENTIL          GABON         1994 2003
>>>   -999   nocode
>>> 6450000    45    942   15 LIBREVILLE/LEON MBA  GABON         1979 2007
>>>    -999    -999
>>> 6450100   -70    875    4 PORT GENTIL          GABON         2003 2007
>>>    -999       0
>>> 6451000   208   1148  599 BITAM                GABON         1971 2007
>>>    -999    -999
>>> 6450300  -342   1065   34 MAYUMBA              GABON         1971 2005
>>>    -999    -999
>>> 6450400   100    960   13 COCOBEACH            GABON         1971 2004
>>>    -999    -999
>>> 6450700  -285   1102   79 TCHIBANGA            GABON         1971 2007
>>>    -999    -999
>>> 6451000   208   1148  599 BITAM                GABON         1971 1996
>>>   -999
>>> 6455100   -72   1023   26 LAMBARENE            GABON         1971 2007
>>>    -999    -999
>>> 6455200    78   1153  583 MITZIC               GABON         1971 2004
>>>    -999
>>> 6455600    57   1287  515 MAKOKOU/EPASSENGUE   GABON         1971 2006
>>>    -999
>>> 6456000   -83   1271  485 LASTOURSVILLE        GABON         1971 1996
>>>   -999     -999
>>> 6456500  -153   1326  573 MOANDA               GABON         1971 1996
>>>   -999
>>> 6455000  -187   1102   89 MOUILA CITY          GABON         2003 2007
>>>    -999       0
>>>
>>> Now, NONE of these stations should contribute to the product, since our
>>> specified cutoff is 75%, or 23 values per month between 1961 and 1990.
>>> But for a couple, a normals line is present:
>>> 6450000    45    942   15 LIBREVILLE/LEON MBA  GABON         1979 2007
>>>    -999    -999
>>> 6451000   208   1148  599 BITAM                GABON         1971 2007
>>>    -999    -999
>>>
>>> These two stations have several things in common. Firstly, they both
>>> have a populated normals line, even though their data counts are too
>>> low. Secondly, their values (and normals) are significantly below the
>>> rest of GABON (ie, between 40 and 180 rather than between 200 and 350).
>>> Thirdly, they both have a swathe of missing values 1997-2002, followed
>>> by patchy data that looks like the rest of GABON (ie, too high for the
>>> normals lines). Here they are, in full:
>>>
>>> 6450000    45    942   15 LIBREVILLE/LEON MBA  GABON         1979 2007
>>>    -999    -999
>>> 6190   65   65   75   87  111  135  149  147  135  110   83   72
>>> 1979-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999   79   83
>>> 1980   59   80   77   85   99  136  149  160  149  100   76   71
>>> 1981   68   59   98   93  125  135  155  156  135   92   83   54
>>> 1982   51   60   73   78  107  147  161  150  149  114   94   70
>>> 1983   79   52   78   93  109  137  182  159  136  105   87   72
>>> 1984   69   62   66   78  109  121  139  154  131  111   95   74
>>> 1985   43   45   72   92  118  123  154  139  139  107   68   83
>>> 1986   64   37   71   82  115  140  146  130  106  113   91   77
>>> 1987   45   65   63  100  100  138  150  149  141  106   83   74
>>> 1988   80   67   76   84  120  136  150  148  134  114   86   93
>>> 1989   76   77   92   84  121  129  156  147  144  116   79   76
>>> 1990   79   85   89   86  112  137  136  151  121  115   84   69
>>> 1991   64   55   97   84  100  119  164  150  138  107   80   68
>>> 1992   66   76   83   93  130  146  153  160  134   95   93   72
>>> 1993   77   61   64   93  110  127  138  127  122   93   64   76
>>> 1994   72   64   91   89  112  133  173  150  137  104  115   84
>>> 1995   71   85   69   95  105  132  178  152  132  130   86   58
>>> 1996   59   60   69   86  101  138  147  153  114  106   89   58
>>> 1997-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 1998-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 1999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2000-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2001-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2002-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2003  305  320  311-9999-9999  275  277-9999-9999  300  303  313
>>> 2004  319  315-9999-9999  290  265-9999-9999  283-9999-9999  307
>>> 2005  316  318  311-9999  291  258-9999-9999  281-9999-9999  299
>>> 2006-9999-9999-9999-9999-9999  295  266-9999-9999-9999-9999-9999
>>> 2007  318  313-9999-9999-9999  277-9999-9999-9999-9999-9999-9999
>>>
>>> 6451000   208   1148  599 BITAM                GABON         1971 2007
>>>    -999    -999
>>> 6190   67   65   74   84  110  135  149  147  133  111   84   72
>>> 1971   66-9999   65   85  122  128  157  149  117  104   79   81
>>> 1972   62   72   73   88  105  121  162  142  114   91   85   70
>>> 1973   69   69   70   78  113  140  150  154  137  100   79   68
>>> 1974   80   72   83   81  106  134  140  151  127   88   85   90
>>> 1975   85   64   76   90  101  129  158  161-9999  100   80-9999
>>> 1976   73   69   61   72  104  132  142  131  128  116   86   60
>>> 1977   70   78   84   77  100  136  149  151  120  120   85   80
>>> 1978   66   62   80   80  117  129  140  132  127  113   82   69
>>> 1979   44   53   74   82  108  143  144  140  127  116   85   83
>>> 1980   60   79   75   77   91  128  145  155  141  101   78   70
>>> 1981   69   59   97   86  120  130  154  159  134  102   90   59
>>> 1982   62   65   72   77  110  150  165  156  150  115   95   71
>>> 1983   82   55   81   91  105  138  181  160  133  107   86   73
>>> 1984   70   63   65   71  106  125  140  154  134  120  100   74
>>> 1985   46   47   70   88  118  124  147  137  137  106   68   83
>>> 1986   66   34   70   77  111  144  147  130  111  119   92   77
>>> 1987   46   66   64   95   98  139  152  153  150  112   87   77
>>> 1988   84   69   78   84  121  135  148  147  130  115   84   92
>>> 1989   77   71   82   76  114  119  145  138  141  116   76   75
>>> 1990   76   81   82   82  107  131  134  150  120  115   86   70
>>> 1991   66   53   88   75   94  117  162  150  137  107   81   68
>>> 1992   65   74   81   84  116  146  166  165  133   89   92   71
>>> 1993   77   63   70  100  123  140  147  135  131  101   68   80
>>> 1994   77   65   83   86  115  133  170  153  136  102  114   85
>>> 1995   71   84   70   96  109  134  178  157  131  134   88   60
>>> 1996   62   58   66   85  103  134  140  149  121  111   83   58
>>> 1997-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 1998-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 1999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2000-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2001-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2002-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2003  255-9999  261-9999-9999  253  250-9999-9999  256  255  259
>>> 2004  265  250-9999-9999  259  245-9999-9999  245-9999-9999  266
>>> 2005  266-9999-9999-9999  257-9999-9999-9999-9999-9999-9999-9999
>>> 2006-9999-9999-9999-9999-9999  253  247-9999-9999-9999-9999-9999
>>> 2007  244  252-9999-9999-9999  253-9999-9999-9999-9999-9999-9999
>>>
>>> I would say that somehow the presence of an illegal normals line is
>>> allowing these 'bad' stations to be included. You'd almost call it
>>> sabotage, wouldn't you?
>>>
>>> The obvious fix would be to replace the normals with missing values,
>>> though it rankles to leave bad data in the database. And we'd end up
>>> with no Gabon stations contributing at all!
>>>
>>>> Sudan: 32.75 E, 13.25 S -- perhaps Khartoum or somewhere just to the
>>>> south of that?
>>>
>>> It's a similar story to Gabon, sorry. This is the culprit, showing
>>> header, normals line, and 2003-2007:
>>>
>>> 6275000  1400   3233  378 ED DUEIM             SUDAN         1971 2007
>>>    -999    -999
>>> 6190   64   60   69   78  102  127  143  143  130  109   83   70
>>> 2003   89   69   63   85  213  210  247  266  247  198  119  102
>>> 2004   87   76   74   86  126  192  212  237  231  168  117  103
>>> 2005   77  107   69   97  125  186  231  247  233  176  108  122
>>> 2006  105   87  103   73  173  179  227  246  239  269   97   83
>>> 2007   80   70   82   95-9999  203-9999-9999-9999-9999-9999-9999
>>>
>>> (again, there was a block of missing values 1997-2002).
>>>
>>> The only other Sudanese stations with enough data to contribute are OK:
>>> 6260000  2182   3148  183 HALFA (WADI)         SUDAN         1971 2007
>>>    -999    -999
>>> 6265000  1917   3048  226 DONGOLA/DUNQULAH     SUDAN         1971 2007
>>>    -999    -999
>>>
>>> In this case, I'd say the 2003-2007 additions to Ed Dueim should be
>>> deleted. Now if we just had a data format with full historical
>>> metadata..
>>>
>>>> I presume you can set these to missing from Jan 2003 onwards?  Or just
>>>> set the outliers to missing?  cc'd to Phil for his suggestion... we
>>>> need something quick and easy, but defendable!  They are so extreme
>>>> relative to the standard deviation that these two particular ones
>>>> cannot really be left as they are.
>>>>
>>>> There are also many other positive and negative vap anomalies
>>>> (especially post 2003, but also some pre 2003) that seem dubious (5 to
>>>> 10 hPa, compared with the vap standard deviation which is not far from
>>>> 1 hPa in most places (well, 0.5 to 2 hPa).  I haven't bothered to list
>>>> the locations/years/months of these strange values because they
>>>> typically last only 1 month and they are mostly in regions of the
>>>> world with more dense observations and hence spatial averaging tends
>>>> to reduce the anomalies rather quickly.  The reason for picking out
>>>> the Gabon and Sudan ones is that they seem to be affecting many months
>>>> and also in regions with less dense networks and hence 1 station
>>>> severely affects an entire national-average (and indeed neighbouring
>>>> countries too).  However I've cc'd this to Phil to get his opinion on
>>>> how much time should be spent on these issues.
>>>
>>> I'd say that if these countries are anything to go by, the VAP database
>>> needs a spring clean!
>>>
>>>> Note that it isn't all bad news :-) There were many dubious looking
>>>> spikes in CRU TS 2.1 vap (even in "well-observed" countries like Spain
>>>> and Portugal) that have improved in CRU TS 3.0 vap!  (see the attached
>>>> time series).
>>>
>>> Cool.
>>>
>>>> Cheers
>>>>
>>>> Tim
>>>>
>>>> P.S.  I've just realised something... shouldn't these anomalies have
>>>> been ignored if they are > 3 (or is it 4) S.D. from the normal?  Or
>>>> maybe that option isn't used for vap?
>>>
>>> As I stated previously - the 3 SD limit's applied by the anomaly
>>> program and so will be in force for both 'real' and synthetic vap.
>>>
>>> Cheers
>>>
>>> Harry
>>>
>>>
>>>>
>>>> At 15:09 21/04/2008, you wrote:
>>>>> Hi Tim,
>>>>>
>>>>> WET almost done; should be finished tonight (will complete it
>>>>> remotely). Note that, if you're investigating it, WET uses a
>>>>> percentage anomalies, I presume this is because the incoming
>>>>> synthetics are PRE-derived.
>>>>>
>>>>> FRS is also within reach, but has a lower priority.
>>>>>
>>>>> Cheers
>>>>>
>>>>> Harry
>>>>> Ian "Harry" Harris
>>>>> Climatic Research Unit
>>>>> School of Environmental Sciences
>>>>> University of East Anglia
>>>>> Norwich NR4 7TJ
>>>>> United Kingdom
>>>>>
>>>> <ncview.vap.dec2004.pdf><vap_cty.pdf><ncview.vap.sep2005.pdf>Dr
>>>> Timothy J Osborn, Academic Fellow
>>>> Climatic Research Unit
>>>> School of Environmental Sciences
>>>> University of East Anglia
>>>> Norwich  NR4 7TJ, UK
>>>>
>>>> e-mail:   t.osborn@uea.ac.uk
>>>> phone:    +44 1603 592089
>>>> fax:      +44 1603 507784
>>>> web:      http://www.cru.uea.ac.uk/~timo/
>>>> sunclock: http://www.cru.uea.ac.uk/~timo/sunclock.htm
>>>>
>>>
>>>
>>
>>
>>
>>
>


