A New Approach For Testing the Accuracy

1
A New Approach For Testing the Accuracy of
Vertebrate Occurrence Predictions
Sandra M. Schaefer William B. Krohn Raymond J.
OConnor Maine Cooperative Fish and Wildlife
Research Unit, and Department of Wildlife
Ecology University of Maine, Orono
2
intro
Testing Wildlife Occurrence Models
Traditionally focused on single species being
predicted on relatively small study areas
3
Habitat Model Testing
Multiple Species
Large geographic areas
4
Traditionally, GAP models are tested by comparing
model predictions to site-specific occurrence
information
Omission Error (OE) failure to predict
species known to occur on the site.
Commission Error (CE) predicting the presence
of species that do not occur on the site.
5
ME-GAP Test Sites
1. North Maine Forestlands 2. Nesowadnehunk
Field, Baxter State Park 3. White Mountains
National Forest 4. Sunkhaze Meadows National
Wildlife Refuge 5. Holt Research Forest 6. Petit
Manan National Wildlife Refuge 7. Rachel
Carson National Wildlife Refuge 8. Moosehorn
National Wildlife Refuge 9. Mount Desert
Island/ Acadia National Park
6
Site-Specific calculations
of species predicted but not present on the
test site Total number of species present on the
site
CE
of species present but not predicted on the
test site Total number of species present on the
site
OE
7
Overall results of accuracy assessment on ME-GAP
predicted species distributions. Medians and
ranges were calculated within taxonomic group
across all sites.
8
Challenges in Testing

• Errors can be caused by multiple factors (i.e
  test site
  
• size, field inventory effort, species
  biology).
• Purpose of the model needs to be considered in
  testing,
• because it may influence how the errors are
  interpreted.
• Interpretation can be complex. Especially for
  commission
• error where the cause may be either apparent or
  actual.

9
Species-Specific Testing Approach
Assessing model accuracy by calculating OE and
CE for each species across multiple sites within
the study area.
10
Species-Specific calculations
of sites where the species was predicted but
not present Total number of potential occurrence
sites
CE
of sites where the species is present but not
predicted Total number of potential occurrence
sites
OE
11
How complete are the field inventories?
Error Range (ER) Difference in the highest and
lowest possible OE and CE. Calculated based on
assumptions of field inventory completeness
(Nichols et al. 1998).
complete assumes that all species occurring on
the site were found during the
field inventories.
incomplete assumes that not all species on the
site were
found during the field inventories.
12
Objectives

• Calculate species-specific ER for avian species
  known to
• regularly breed in Maine.
• 2) Determine if there is a relationship between
  the ER
• and the extent of a species distribution.
• 3) Determine if there is a relationship between
  the ER and
• how likely a species is to occur during a
  field survey
• (for statewide species).
• 4) Compare the test results from the
  species-specific and the
• site-specific approaches.

13
MethodsCalculating the species-specific ER
14
Potential Occurrence
Species could potentially occur on a site if the
site is within the range limit. Field survey
data was also included if it indicates the
species occurs on the site.
15
Species Occurrence Table
1 Within ME-GAP Range P Presence A
Absence
16
MethodsData Analysis
17
1) Spearmans Rho used to test for a relationship
between a species distribution and the
commission ER.
2) Spearmans Rho used to test for a relationship
between an a a priori ranking system called
Likelihood of Occurrence Ranks and the
commission ER.
3) Site-specific commission error results
compared to the species-specific error
ranges. Grouped into primary
breeding category (i.e., barren, early
successional forest coniferous, forest
deciduous, forest generalists, and wetland).
Mean error for the site-specific method was
plotted against mean error for
species-specific method. One-way ANOVA was
used to determine if a significant difference
existed between the means of the 2 methods.

18
Likelihood of Occurrence Ranks (LOORs) (Boone and
Krohn 1999)
LOORs are an a priori system of ranking species
based on how likely they are to be seen during a
standard wildlife inventory.
Developed to help interpret the causes of
commission error. (Schaefer and Krohn 2002).
Atlas occurrence information was used to generate
a spatial incidence for each species. The
incidence came from dividing the number of
survey blocks in the atlas having confirmed or
potential breeding by the number of survey
blocks within the species range.
19
Results
20
Assumes Complete Field Survey
Assumes Incomplete Field Survey
A.
B.
Number of Species
Commission Error
Commission Error
C.
D.
Omission Error
Omission Error
Frequency distribution of OE and CE from
predicted avian occurrences with the assumption
of complete (A and B) and incomplete (C and D)
field survey data.
21
C
B
A
Frequency distribution of the number of test
sites on which each bird species could
potentially occur based on range limits in
Maine. A those species with limited
distribution, B those species that are
moderately distributed, and C those species that
are statewide.
22
Commission Error Range
Number of potential occurrence sites
Commission ER for all avian species across all
test sites. rho -0.583 P lt 0.001
23
Commission Error Range
Likelihood of Occurrence Ranks
Relationship between CER and the LOORs. rho
-0.657 P lt 0.001
24
Means and 95 confidence interval of CE, by major
breeding habitats, for the site-specific (light
bars) and the species-specific (dark bars)
approaches to testing predicted avian
occurrences.
25
Conclusions

• Calculating species-specific ER provided an
  opportunity
• to assess the overall predictive quality of
  the habitat models, as well
• as determine the variability of error for each
  species.
• CER was significantly correlated with species
  distribution as
• well as with how likely a species was to be
  observed on
• a field inventory.

26
Conclusions (cont)

• If a high ER is reported for a species that has
  a high likelihood
• of occurrence then the most likely cause for
  the over prediction is
• in the model.
• However, if a species has a low likelihood of
  occurrence
• and a high ER, then the over prediction error
  is likely due to
• having incomplete field surveys for the
  species.

27
Summary

• Both methods are influenced by the data available
  for use in the
• testing process.
• The site-specific method provides a generalized
  idea of how well
• the models are capturing species presence and
  absence
• and across the entire state.
• The species-specific approach gives a more
  detailed description
• of which species are reporting the highest
  levels of error.
• This helps to answer the question of why the
  predictive error
• is being reported.

28
Take Home
I recommend using both methods in assessing
model accuracy because the two approaches
provide different information about the quality
of predicted occurrences.
29
Acknowledgements
William B. Krohn Raymond J. OConnor Daniel J.
Harrison Steve R. Sader Randall B. Boone William
Haulteman
Supporting Organizations and Agencies Gap
Analysis Program, USGS BRD Maine Cooperative Fish
and Wildlife Research Unit Department of Wildlife
Ecology University of Maine