COMPARISON OF VALIDATION PROCEDURES TO DETECT MEASUREMENT ERRORS IN AN AREA FRAME SAMPLE SURVEY

1
COMPARISON OF VALIDATION PROCEDURES TO DETECT
MEASUREMENT ERRORS IN AN AREA FRAME SAMPLE SURVEY
Laura Martino, Marco Fritz, Marjo Kasanko
Javier Gallego
European Conference on quality in official
statistics, Rome 8-11th July 2008
2
Contents

• Scope of the work
• Methods for measurement errors detection
• The case study LUCAS survey
• Conclusions

3
Scope of the work

• The scope of the work is comparing the efficacy
  of three selected procedures to detect
  measurement errors in an area frame sample survey
  that uses a combination of orthophoto
  interpretation and ground survey

4
Methods for detection of measurement errors

• Three methods have been selected to be compared
• double-blind survey
• check of ground data with ausiliary data sources
  
• cross-check of orto-photo interpretation and
  ground-survey outcomes to evaluate the accuracy
  of stratification

5
Case study Land Use and Coverage Area frame
sample Survey (LUCAS)

• Area frame sample survey carried out by Eurostat
  since 2001.
• Main objectives are providing
• coherent and harmonised statistics on land use
  and land cover
• b) a common sampling base (frame, nomenclature,
  data treatment) to be used for further scopes
• c) ground evidence for calibration of satellite
  images
• d) information on aspects relating to the agro-
  environment

6
Second phase sampleGround survey

• During the ground survey, surveyors
• Reach the point using GPS and cartographic
  material

• Record land cover and land use according to the
  full nomenclature

• Take pictures (Point, Cov,N,E,S,W,irr)

• Walk the transect (250 m. East)

7
1 procedure Double blind survey

• A second survey was organised in 2006 in
  parallel to the main LUCAS 2006 survey.
• It was conducted by an independent company not in
  charge of the LUCAS survey in the countries
  involved.
• No information were provided on the results of
  the main survey.
• The double blind-survey sample size represented
  5 of the total survey sample and counted almost
  8200 points.

8
Outcomes of double-blind survey

• The possible outcomes of the double-blind survey
  were
• Land Cover/Land Use being the same between the
  two surveys (correct points observed from the
  same location)
• Land Cover/Land Use being different between the
  two surveys but both correct (different rules of
  observation - look north, east, etc. - or change
  in the land cover between the two visits (crops
  harvested, sown, building built, etc...)
• Land Cover/Land Use being different between the
  two surveys and lack of sufficient information to
  say which survey is correct
• Land Cover/Land Use being different between the
  two surveys and the double-blind survey being
  correct
• Land Cover/Land Use being different between the
  two surveys and the main survey being correct.

9
Double blind survey
D-B survey 8,200 pts
Main survey 169,000 pts
D-B survey introduced as many errors as the main
survey
10
2 procedure Use of ausiliary data to correct
results

• IACS data
• Pictures
• Other georeferenced information

Large improvement in data quality through the use
of pictures
11
3 procedure comparison of orthophoto
interpretation and ground survey

• A comparison of the classification of the points
  according to the ground observation and the
  photointerpretation has been conducted in a
  comparable nomenclature of 7 classes.
• A proportion of agreement has been computed
• Since points belonging to different strata have
  been subsampled with probability that could be 5
  times larger/lower, a weighted proportion of
  agreement is computed in addition to the
  unweighted one.

12
(No Transcript)
13
Measurements of agreement

• Frequency The unweighted proportion of agreement
  is 70.8, while the weighted agreement is 74.8.
• Kappa index (Bishop et al., 1975)
• where are the proportions of each
  cell of the table
• The unweighted Kappa is 0.58 (0.66 if we take
  into account the strata weights).

14
Rate of agreement
15
Mainly confused items

• Some points that are agricultural land are
  photo-interpreted as non-agricultural for several
  reasons location inaccuracy, land cover change
  or simply photo-interpretation errors
• The main confusions occured between
• permanent grass and bare land are often
  photo-interpreted as arable.
• Woodland or shrubland interpreted as grass. This
  may be partly due to problems to apply in
  practice the definition of forest or woodland

16
Conclusions

• The pre-eminent result of the double blind survey
  was that it introduced as much errors as the main
  survey for many reasons
• The analysis of pictures and the use of auxiliary
  information appeared to be fundamental to detect
  measurement errors and improve data quality
• Orthophoto interpretation produce some
  inefficiency (stratification accuracy) -gt
  poststratification???

17

• THANK YOU FOR YOUR ATTENTION!!!

18
Comparison with Corine Land Cover 2000
200 metres
Mapping based on satellite images and orthophoto
interpretation
19
LUCAS and CLC
20
Visual metadata