Title: Multitemporal SAR image segmentation and classification based on a semanticbased stratified SAR and
1Multi-temporal SAR image segmentation and
classification based on a semantic-based
stratified SAR and optical data fusion approach
- François Aspert, Ecole Politechnique Federale de
Lausanne (EPFL) -SPI - Francesco Holecz, Sarmap
- Andrea Baraldi, JRC
- Stefano Natali, MEEO
- Jean-Philippe Thiran, EPFL-SPI
2Content
- Technological scenario in SAR remote sensing
imagery - Image segmentation problem definition in
biological vision - Goal of this work
- Methodology
- SAR and optical data pre-processing requirements
- SAR data edge preserving smoothing Edge
detection Standard region growing segmentation
employing an edge map (Buhmans). - Preliminary spectral-based classification as an
auxiliary source of semantic information. - Multi-source semantic-based stratified
segmentation (applied to the standard Buhmans
segmentation algorithm for comparison purposes). - Conclusion and future works
3Technological scenario in SAR remote sensing
imagery
- Currently, 8 spaceborne SAR systems are
operational. By the end of 2008, 3 additional SAR
sensors are planned to be launched, thereby
allowing to obtain an almost continuous
monitoring of the Earth coverage at large scale. - In order to transform the huge amounts of
multi-temporal multi-source SAR and Optical data
into information, automated (either prior
knowledge-based or unsupervised) data
pre-processing and image understanding
(classification) techniques are mandatory, as
ground truth data and ancillary information are
typically tedious, expensive, and either
difficult or impossible to gather.
4Biological vision
- Biological vision consists of a set of ill-posed
problems, such as (Victor, 1994) - Shape-from-shading finding boundaries between
regions of uniform but different
intensity/spectral values. - Shape-from-texture finding boundaries between
regions of uniform but different textures. - Structure-from-motion.
- Figure-from-ground.
- The visual system necessarily makes inferences
fom these partial information, and the discovery
of how these inferences work, is what the study
of bological vision is all about.
5Image primal sketch in computer vision
- Image segmentation (simplification) Preliminary
image mapping Image primal sketch (Marr,
Vision, 1980). - Provide an image partition into asemantic areas
(regions, segments, objects, e.g., segment1,
segment2, etc.) of uniform either i)
intensity/spectral values (shape-form-shading),
or ii) texture (shape-from-texture). - The intrinsic vagueness of the abovementioned
term uniform reflects the inherent
ill-posedness of the image segmentation problem. - The dual problem of an ill-posed image
segmentation task is the ill-posed edge
detection, either i) intensity/spectral, or ii)
textural.
6Image segmentation in computer vision
- In practice, i.e., in real-world commercial
software toolboxes, image segmentation algorithms
provide multi-scale solutions in the hope that
the target phenomenon will appear correctly
segmented at some scale. - Unfortunately, this operational approach is
contrary to the principle of the visual
perception of texture (Julesz, 1983) The
aperture of attention changes in spatial scale
according to the size of the feature being
sought. ? Each picture primitive component is
represented only once in its true form to be
combined into complex objects as spatial
arrangements of basic component parts
(recognition-by-components).
7Goal of this work
- To propose an original set of algorithms aiming
at the automated multi-temporal SAR and optical
image data fusion, simplification (information
aggregation, e.g., asemantic segmentation, if
any), and classification.
8Comparison of a traditional SAR Optical
segmentation data fusion algorithm vs. a
stratified segmentation approach
Conceptual inconsistency???!!! Optical
data-driven information should be employed only
once at a symbolic (semantic) level in an early
stage of the SAR segmentation workflow. Goal of
this experiment comparison purposes!!!!
Preliminary classification means preliminary to
(morphological, tetxural, geometric, etc.)
feature extraction!!!!!!
SAR data pre-processing
- Focusing - Geometric Calibration -
Radiometric Calibration - Radiometric
Normalization
Digital Elevation Model
Radiometrically calibrated optical data
Multi-temporal Anisotropic Diffusion Iterative
edge preserving smoothing of fused data at the
asemantic pixel level
Data fusion at the asemantic pixel level
(stacking) Edge preserving smoothing Edge
detection
Brightness image generator
Edge Detection
Standard region growing segmentation employing
an edge map
Seg.
Classification map
Object-based classification
Object-based classif.
9SAR/optical data pre-processing requirements
- Due to the increasing number of SAR and optical
sensors, it is conditio sine qua non that
automated pre-processing supports a pletora of
different sensor imagery acquired in different
geometries, polarization, etc. As a consequence,
temporal, spatial, (and spectral) project
requirements can be optimized. - To allow automated data understanding, the input
dataset must be well behaved, i.e., SAR and
optical images must be i) radiometrically
calibrated and ii) geometrically corrected (not
the other way around!!).
10ENVISAT ASAR Pre-processing
ENVISAT ASAR image mosaic of Senegal consisting
of about 180 multi-temporal images geometrically
and radiometrically pre-processed in a fully
automated way.
11Intensity/Color-based Multi-temporal Anisotropic
Non-Linear Diffusion (MANLD) for non-parametric
SAR speckle removal
Perona and Malik, 1990 Acton and Landis, 1997
- No texture model is adopted, i.e., no texture
segmentation ispursued, but intensity/color-based
region growing (diffusion). - Use the heat diffusion equation for iterative
edge-preserving smoothing employing no speckle
(multiplicative noise) model. - The diffusion coefficient c 1 in the interior
of each region (where gradient ?I 0), whereas c
-gt 0 at each edge point. - k image-wide data-driven free parameter (edges
with gradient lt K are not smoothed).
12Multi-temporal Anisotropic Non-Linear Diffusion
(MANLD)
Original ENVISAT ASAR HH image
MANLD smoothed 4-temporal ENVISAT ASAR HH image
sequence
13Multi-temporal Anisotropic Non-Linear Diffusion
(MANLD)
MANLD smoothed 4-temporal ENVISAT ASAR HH image
sequence stacked with one Optical image
(pixel-based data fusion)
Original ENVISAT ASAR HH image
14MANLD for edge-preserving smoothing
- Advantages
- Only one free parameter k which is image-wide
data-driven (Acton and Landis, 1997). - Edge-preserving smoothing is achieved.
- Advantage using multi-sensor time-series (SAR and
Optical) vs. single-date image diffusion. - Limitations
- Inability to reject outliers (sensitive to noisy
data). - Creation of artifical edges (called staircase
artefacts).
15Edge detection in cascade to edge-preserving
smoothing
- First-stage edge-preserving smoothing by MANLD
allows in cascade - Effective edge detection (high sensitivity).
- Good edge localization.
- Clear (1!) response per edge.
- Existing edge detection techniques
- Canny, Prewitt, Sobel, Marr-Hildreth, etc.
- Canny edge detector is optimal with respect to
the abovementioned criteria.
16Edge Detection
F. Aspert et al., ENVISAT Conference, March 2007
- Use Canny to get optimal edge detector
- Smoothing
- Edge magnitude
- Edge direction
- Non-maximum suppression
- Hysteresis thresholding
- Extends Canny edge detector
- Use First Fundamental Form multi-temporal
gradient definition - Find multi-temporal gradient magnitude
- Find multi-temporal gradient direction
17Edge detection - Multi-temporal vs Single-date
edge detection
Single-date edge map, Input dataset 1 ENVISAT
ASAR image
Multi-temporal edge map, Input dataset 4
ENVISAT ASAR and 1 Optical image
First-stage edge preserving diffusion algorithm
Automatic diffusion coefficient estimation, 60
iterations. Second-stage Canny edge detection
Automatic threshold computation.
18Standard region growing segmentation employing an
edge map
l. Hermes and J. Buhman, IEEE TGRS, Sept 2006
- Buhmans region growing is a two-stage image
segmentation approach - Minimize an objective function to find seed
segments in the whole image sequence. - Use statistical properties to promote large
regions bounded by edges belonging to an edge
map. - User-defined parameter
- Region merging stopping criterion maximum size
of a region (set equal to 1500 pixels, based on
heuristics!!!).
19Standard region growing segmentation employing an
edge map
Multi-temporal segmentation featuring 200 output
regions. Maximum region size (stopping criterion
based on heuristics) 1500 pixels.
MANLD filtered image with automatic sensitivity
thresholding and 60 iterations.
20Standard region growing segmentation employing an
edge map
- Disadvantages
- Stopping criterion based on heuristics.
- Large amount of memory required.
- Large processing time.
- Advantages
- Edge map improves final segmentation
21Multi-source semantic-based stratified
segmentation approach
- Rationale
- Optical and SAR data feature complementary
surface type investigation properties. For
example, in vegetation land cover detection - Optical data is suitable for separating
vegetation, ranging from low (e.g., rangeland) to
large biomass (e.g., forests), from
non-vegetation. - SAR data is suitable for detecting forest stands
(based on image texture properties), but it is
little sensitive to grassland and rangeland. - Preliminary spectral classification is adopted to
pursue semantic-based stratification of any
standard segmentation algorithm. - Segmentation is performed stratum specific,
according to a well-known divide-and-conquer
problem solving approach capable of making the
inherently ill-posed image segmentation problem
easier to solve.
22Fully automated preliminary spectral
classification - Optical single-date imagery
A. Baraldi et al., IEEE TGARS, Sept 2006
WR DR PBHNDVI PBMNDVI PBLNDVII BBBHTIRF BBBHTIRNF
BBBLTIRF BBBLTIRNF SBBHTIRF SBBHTIRNF SBBLTIRF SB
BLTIRNF ABBHTIRF ABBHTIRNF
ABBLTIRF ABBLTIRNF DBBHTIRF DBBHTIRNF DBBLTIRF DBB
LTIRNF SHB WS TW TKCL TNCL SHCL SN ICSN TWSHSN SU
Unclassified SVHNIR SVLNIR AVHNIR
AVLNIR WVHNIR WVLNIR WE SHV SSRHNIR SSRLNIR ASR
HNIR ASRLNIR SHR AHR
Single-date optical Data
Preliminary spectral knowledge-based classifier
SOIL MAPPER
Classification map
SOIL MAPPER by MEEO provides an exhaustive and
mutually exclusive semantic image partition into
six spectral categories i) Vegetation, ii) Bare
soil/Built-up, iii) Water/Shadow, iv) Snow/Ice ,
v) Clouds, and vi) Outliers.
23Fully automated preliminary spectral
classification - Optical single-date imagery
24Fully automated bi-temporal post-classification
land cover change detection
Sarmap, 2007
Optical, Time 1
Optical, Time 3
Optical, Time 4
Optical, Time 2
25Fully automated bi-temporal post-classification
land cover change detection
Data Landsat-5 TM
26Traditional multi-source segmentation vs
Semantic-based stratified multi-source
segmentation - Model comparison
- Dataset
- One SPOT-4 image acquired on September 2006
- Four ENVISAT ASAR HH/HV images acquired between
May and August 2006 - Stratified segmentation is performed exclusively
on agriculture candidate areas. Objective
identify the development of agricultural fields
through the May-August 2006 time period. - Stratified parameter setting in the standard
region growing algorithm (Buhmans) - Edge preserving diffusion algorithm Automatic
sensitivity thresholding, 60 iterations. - Edge detection Automatic threshold computation.
- Region merging The maximum size of a region has
been set to 1500 pixels.
27Multi-source semantic-based stratified
segmentation approach
Preliminary spectral classification map generated
from the single-date SPOT-4 image. Only the
agriculture candidate area (as an OR combination
of 3 spectral classes) is considered for the
stratified segmentation.
28Multi-source semantic-based stratified
segmentation approach
Stratified segmentation (Buhmans) where data
fusion occurs at the semantic pixel level
Traditional segmentation (Buhmans) where data
fusion occurs at the asemantic pixel level
29Multi-source semantic-based stratified
segmentation approach
- Advantages
- Consistent with any traditional segmentation
approach. New combination of well-known image
processing algorithms!!!!!!! - Less memory occupation.
- Less processing time.
- Stratum-specific parameter setting
(divide-and-conquer approach). - Increased sensitivity (level of genuine but small
image details detected) . - Increased accuracy.
- Encourage data fusion at the pixel-specific
semantic (symbolic) level rather than at the
level of unlabeled asemantic digital numbers. - Future developments
- Clever exploitation of the stratified
segmentation approach, i.e., employ spectral
driven semantic information only once in the
early stages of the SAR data procesing workflow. - Preliminary rule-based classification of SAR
imagery.
30Comparison of a traditional SAR Optical
segmentation data fusion algorithm vs. a
stratified segmentation approach
SAR data pre-processing
Optical data pre-processing
- Focusing - Geometric Calibration -
Radiometric Calibration - Radiometric
Normalization
- Radiometric calibration into TOA
reflectance - Orthorectification -
Topographic correction
Digital Elevation Model
Digital Elevation Model
Preliminary spectral classifier SOIL MAPPER
Preliminary rule-based classifier
Primal sketch in the Marr sense
Primal sketch in the Marr sense
Data fusion at the semantic pixel level
Multi-temporal multi-source Data Fusion at the
semantic pixel level (fuzzy
rule-based) classification
Preliminary spectral-based classification map
Preliminary rule-based classification map
SAR Optical data classification map
31Preliminary prior knowledge-based classification
- Interferometric SAR
Sarmap, 2008 A. Roth and T. Esch, DLR, 2007???
Interferometric Dual Polarization SAR Data
Unclassified Stable Smooth Surface - Water
Road Unstable Smooth Surface - Water Road Dense
Vegetation Average Vegetation Short Dry
Vegetation Unstable Short Vegetation -
Decrease Unstable Short Vegetation - Strong
decrease Unstable Short Vegetation -
Increase Unstable Short Vegetation - Strong
increase Average Vegetation Change -
Decrease Strong Vegetation Change -
Decrease Flooded Vegetation Average Vegetation
Change - Increase Strong Vegetation Change -
Increase Weak Vegetation Change - Vegetation
loss Weak Vegetation Change - Vegetation
growth Stable Short Vegetation Stable Bare
Soil Unstable Bare Soil Stable Very Dry Bare
Soil Stable Very Wet Bare Soil Strong Scatterer -
Urban Rock Drying Flooding
Knowledge based Classifier
25 Classes
32Preliminary prior knowledge-based classification
- Interferometric SAR
Data Interferometric ALOS PALSAR HH/HV Dual
Polarization (June-Aug 2007)
Preliminary classification map
33Preliminary spectral classification - Optical
single-date imagery
Data Landsat-5 TM (October 2000)
Preliminary spectral classification map