A Measurement Error Calibration Experiment in a Longitudinal Study

1
A Measurement Error Calibration Experiment in a
Longitudinal Study

• Jason Legg
• Center for Survey Statistics and Methodology
• Iowa State University
• at Amgen Inc. on June 24th, 2008

2
Longitudinal surveys are often designed to
measure change

• Same units observed
• Within unit correlation
• Observation schedule
• Examples
• Change in patient conditions after treatment
• Spread of an infectious disease
• Change in land use related to government policy
• Observations through devices, technicians,
  protocols
• Measurement error (ideally time correlated)

3
Measuring tools and procedures change over the
course of long studies

• Reasons
• Reduce measurement error
• Cost savings
• Ease data collector burden
• Decrease nonresponse
• Two goals for a longitudinal dataset
• Reproducibility of previously released estimates
• Estimate change on the same variable
• Solution
• Calibrate new method to the target of the old
  method

4
Calibration with a gold standard known x
Regress Y on x


5
Ordinary least squares is biased if x is measured
with error
Regress Y on X


6
Our measurement change is for determining
developed land in the NRI

• National Resources Inventory
• Status and trend on nonfederal land
• Longitudinal panel survey
• Agricultural/environmental emphasis
• Land use classification
• Wetlands
• Management practices
• Erosion
• Data Collection
• Photograph interpretation
• View and potentially update previous years

7
NRI has a two-stage stratified design
Collect areas for developed land, transportation,
and water bodies at the segment level
8
Goals of the experiment

• Determine if procedures are sufficiently
  calibrated
• Estimate the overall improvement from the new
  procedure
• Estimate the measurement error variance functions
• Useful as metadata
• Can be used to adjust bias in regression

9
Old procedure delineate areas

• Transparency on image with light desk
• Delineation of segment area polygons using a
  planimeter
• Use historical photographs, overlays, and
  measurements

10
New procedure digital delineation and residence
marking

• All photographs are digitized
• Digitally delineate
• Nonresidential urban areas
• Roads
• Streams
• Waterbodies
• Mark each house roof with a
• Computer program converts residence marks to
  areas
• Hexagon for each house
• Linking rules

11
Example polygons under the old protocol
12
Example polygons under the new protocol
13
Removed subjectivity of residential areas

• More repeatable than delineation
• Difficulty determining yard boundaries
• Roads and nonresidential areas are delineated as
  before
• Less uniformity in size
• Program parameters
• Distance to link
• Number of houses needed to contribute
• Hexagon size

14
Calibration experiment data are triples

• All segments have old protocol data
• Old protocol collection sequence
• Experiment segments have new protocol data
• Two data sequences for the new protocol
• Intermediate collectors reduce 2003 measurement
  error correlation

1997
2001
2003
2001 Person A
2003 Person B
1997
2001 Person C
2003 Person D
15
Control for data collector effects

• 8 data collectors grouped
• 8 segments given to each group at a time
• Collectors randomly divided into 2 groups of 4
• Latin Square to assign collection type
• Pool the 8 data collectors after each 8 segments
• Some additional control to ensure group mixing

16
Segments were selected that contained interesting
features

• Based on the 2003 observations under the old
  protocol
• Exclude 100 urban, water or federal segments
• Include all segments with urban or water
  2001-2003 change
• Sample
• With high rate segments with developed land
• With low rate segments with water and no
  developed land
• With extremely low rate segments without water
  and developed land
• Biased selection
• 503 sample segments for analysis

17
Calibrate using the proportion of developed land
in a segment

• Notation

18
A slope shift detection calibration model
Target of old procedure
?0 0, ?1 1, and ?2 1 is calibrated
Sample moments
19
Estimate ?i using a proxy for xi from a simpler
model
20
The simple model is moment saturated

• Transform observations
• Estimate the sample covariance
• Method of moments

21
An EGLSE for xi is constructed using the simple
estimates

• Regress on
• Like factor scoring or first step in two-stage
  least squares

22
The split point model is fit conditional on the
estimated split

• Write the mirror regression model
• Easier to correct bias
• Regress Xi on

23
Y1i -Y2i can be used to correct the bias in the
regression

• Ordinary least squares denominator DD contains
  terms
• Bias corrected regression

where
and A1 indexes and A2 indexes
24
Test for calibrated versus split line

• Test
• F0.52 on 3 and 497 degrees of freedom
• P-value 0.67
• Retain calibrated hypothesis

versus
25
(No Transcript)
26
Some misfit near a proportion of 0

• Bin 1 and 2 data are not related to housing units
• Effect is small on total estimates

27
Variance function estimation after calibration
(xi yi)

• Two variance response variables
• Modeling assumptions
• , proportionality
• Symmetry of variance functions around x 0.5
• Equal difficulty measuring 40 developed as 60
• Variance of
  is proportional to
• Constant coefficient of variation model

estimates
estimates
and similarly for
28
Data are transformed due to extreme values

• Measurement errors are very right skewed
• Generalize least squares fails for highly skewed
  data
• Behave like squares of chi-square variables
• Square root transformation
• Fitted models
• 2.5 power comes from fit and properties of the
  MLE for multiples of chi-square variables

29
Solve using generalized least squares

• Gauss-Newton algorithm
• Initial values of ? and gi were given
• Equations weighted by inverse of previous fit
• Constant coefficient of variation
• Ratio to transform back to square scale

30
Estimated parameters of the variance functions

• Similar ratio of variances as in the other
  analysis
• Can use variance functions to refit the
  calibration models
• F-test result gives a similar conclusion to that
  of the t-tests
• Small bias in parameter estimates

31
? ?
Standardized Squares
32
Discussion

• Use of the structure in other areas
• Change in medical devices
• New lab technicians
• Does a gold standard fix the problem?
• When is procedure considered calibrated?
• Other approaches
• Instrumental variables
• Reliability and Reproducibility experiments

33
Thank you! Questions?