SAFE SOFTWARE The Spatial ETL Experts

1
v
Sep. 21-22, 2006
The Road Ahead
2
Housekeeping

• Wireless Internet Access
• UserName hccxx
• Password 55tt66yy
• Lightning Talks start at 100 PM.
• Conference Website www.safe.com/2006uc

3
Doctors Report
4
FME 2007 and Beyond

• FME 2007 Overview
• Product Integrations
• Python in FME 2007
• FME 2007 Spatial ETL Server debut
• Demonstrations

5
FME 2007

• New Geometry
• Architectural refresh continues focusing on
  processing functions.
• 64 bit FME engine is running internally.
• Not all formats as some rely on 3rd party
  libraries.
• Cannot have mixed 32 and 64 bit in an application
• Raster
• Improved performance. Reprojection is about 40
  times faster.
• More Web Service clients
• WMS, WCS, GeoRSS and others.

6
FME 2007

• More Formats
• Geoconcept, GeoRSS, VOIV, CityGML, Lidar, TIGER
  GML., File Geodatabase
• Format updates
• Oracle Spatial updates, Enterprise Geodatabase,
  Autocad, DGN, GeoMedia,
• Continued Workbench Improvements
• Overview window, increased window flexibility
• Improved do/undo
• Improved schema migration support
• Improved transformer upgrade support.

7
Workbench 2007
8
Workbench 2007
9
Workbench 2007
10
Workbench 2007
11
Customizable Categories

• Create and share groups of transformers
• Transformers could grouped by task

12
Selectable Reproject Engine

• Default is FME engine
• ESRI Reproject Engine can be used if ArcGIS is
  present
• Possibly more Reproject engines could be made
  available
• An FME Objects interface for Reproject engines
  has also been added
• Special license needed

13
Raster Support Goals

• Translate between raster types
• eg. GeoTIFF gt JPEG2000
• Translate between raster subtypes eg. 8 bit
  colour palette -gt 24 bit continuous colour
• Resample rasters
• Reproject rasters
• With reasonable performance(!)
• Combine Raster Vector in interesting ways
• Clip rasters to specific vector rectangles
• Drape -- moving raster data to vector features

14
Raster in Databases

• Raster database support status
• ESRI ArcSDE
• Writing and reading
• Oracle 10g
• Writing (reading will follow)

15
Zoning and Roads Overlaid on Landsat Image
16
Contours Generated From DEM
17
Why 64 Bit FME?

• Memory, memory, memory.
• Able to process drastically larger datasets.
• Machines with more than 4 GB commonplace.
• Increased processor performance
• No cost premium
• Vista Core Duo 2 commonplace 64 bit

18
New Geometry Model
19
New Geometry Model

• Backward compatible with old geometry model
• Old geometry is in fact a type of new geometry.
• All existing formats/transformers work as before.
• Formats upgraded as needed to support new
  geometry
• ESRI Geodatabase, FF, GML, Oracle, AutoCAD,
  GeoMedia, Design Files, SDF3,
• Enables FME to support all latest geometry models
• Latest status on fmepedia.

20
Why a New Geometry Model?

• Arcs, Arcs, Arcs
• Many systems represent features using arcs.
• Paths can consist of both arcs and line segments.
• More efficient/accurate way to represent curves.
• New CurveFit transformer will create paths with
  arcs.
• Proper support for measures.
• Multiple measures per vertex also supported.
• Lays groundwork for multiple geometries per
  feature.

21
Multi-Processing

• Cores, Cores, Cores.
• Intel/AMD now have dual core processors.
• How to take advantage of this?
• Multi-threading applications
• Multi-processing applications
• Advantages of Multi-Processing
• Each process has its own memory space.
• Multi-process workspace can be distributed across
  machine boundaries not just process boundaries.

22
Multi-Processing

• Disadvantages of Multi-Processing
• Communication is more expensive.
• Big performance gain for high cpu intensive tasks
  like intersection, overlays, surface modeling,
  etc.
• Advantages of Multi-Threading
• Low communication overhead.
• Finer grained parallelism.
• Disadvantages of Multi-Threading
• More difficult to debug and develop applications.
• Safe has Multi-Processing via the FME
  TransporterFactory Transporter Transformers

23
More Data Sources

• FME is an environment able to use web services.
• To FME these are just data sources, and
  transformers.
• FME is an ideal data integration/aggregation tool
  able to consume many data sources.
• It doesnt matter where the data comes from!
• Python plays critical part in this.
• Our coming FME Spatial ETL Server can publish
  services derived from other services.
• FME is positioned to play a key role in enabling
  organizations to make wide-spread use of web
  services.

24
Python

• Python is a great way to extend the reach of FME
  technology.
• Most web services are not WFS, WMS, or SOAP..
• FME 2007 enables users to exploit the power of
  Python more easily.
• FME 2006 GB has some capabilities but not like
  FME 2007.

25
Python Goodies

• Many great libraries out there for users
• PyParsing (generic parsing)
• BeautifulSoup (XML/HTML parsing)
• Cheetah (Template Library)
• ReportLab ( PDF generation, etc)
• ZSI (Soap library)
• CElementTree (Wicked fast XML parsing)
• GeometryFactory (geometry algorithms)
• Many more

26
Web Service via Python

• !

Call out to Yahoo Geocoding Web Service
27
FME Spatial ETL Server 2007

• FME has very strong client applications
• Workbench is a great authoring tool
• Viewer is a strong data inspection tool for
  raster and vector
• Workbench is to be the authoring environment for
  Server
• Deploy workbench scripts to the server as a
  service/resource for organization
• Run/debug workbench scripts on the server during
  the authoring phase
• SpatialDirect is a client to this FME Spatial ETL
  Server
• FME Spatial ETL Server will be released at same
  time as FME 2007
• It will then evolve over the next few releases.

28
Spatial ETL Server Solution
FME Spatial ETL Server
29
FME SpatialDirectConfiguration
FME Spatial ETL Server
30
FME Spatial ETL Server
FME Spatial ETL Server
31
FME Spatial ETL Server
FME Spatial ETL Server
32
FME Spatial ETL Server
Users see data the way they need to see it
33
Spatial ETL Server
Ability to do data Federation is key
34
Survey at ETL Conference

• What is the most important data type that ETL
  tools dont support?
• 2 answer was Spatial!

35
MIS GIS IS

• Databases
• All major databases both open source and
  proprietary now support or will support spatial
  data natively.
• ETL Tools
• Many of the Major ETL vendors such as Microsoft
  (SSIS) and Informatica now want to incorporate
  spatial capabilities.
• Business Intelligence
• Many decisions making processes can benefit from
  incorporation of spatial data.
• Internet
• Yahoo Maps, Google Maps, Google Earth, Microsoft
  Virtual Earth, AOL, GlobeXplorer,

36
Safe and Microsoft SSIS
37
Safe and Informatica PowerCenter
38
Safe Software Strategy

• There is one thing that all applications need to
  shine.
• Spatial Data
• Data is to applications and databases as fuel is
  to an engine.
• A database or web-based tool with no data is
  useless.
• Web services with no easy way to consume them
  provide no value.
• The job of Safe Software and its Users is to
  bring the power of spatial data to all
  applications!

39
Demonstration(s)

• Custom Transformers (fmepedia)
• New Geometry
• MRFClean, and CurveFit
• Python
• SSIS
• Spatial ETL Server

40
Demonstration(s)