WMO STRATEGY FOR MIGRATION TO TABLE DRIVEN CODE FORMS

1
WMO STRATEGY FOR MIGRATION TO TABLE DRIVEN CODE
FORMS

• 13th EGOWS meeting
• Rome, 10-13 June 2002
• (Joël Martellet,
• WMO, World Weather Watch, Data Processing
  Systems)
• EXPERTS CBS Expert Team Dr Fred Branski (USA,
  Chairman), Jean Clochard (France, Chairman Expert
  Team on Codes), Seid Amedie (Ethiopia), Heinrich
  Knottenberg (Germany), Dick Blaauboer
  (Netherlands), Dr Vladimir Antsypovich (Russia),
  Milan Dragosavac (ECMWF), Dr Simon Elliott
  (EUMETSAT), Etienne Charpentier (JCOMMOPS)

2
OUTLINE

• THE SITUATION IN 2002
• WHY SWITCH TO TABLE DRIVEN CODES ?
• A MIGRATION WITH CONSTRAINTS
• IMPACTS ON WORLD WEATHER WATCH DATA FLOW AND
  ASSOCIATED PROGRAMMES
• ACTION PLAN
• TENTATIVE MIGRATION SCHEDULE
• SOFTWARE HOUSE PROJECT

3
Observations data flow in 2002

• For the operation of the World Weather Watch,
  traditional meteorological and marine in-situ
  observations are still exchanged between
  Countries in Traditional Alphanumeric Codes (TAC)
  (although their total volume is only several
  megabytes (about 645 MB for BUFR satellite data)
  per day, it is still (for many years) a required
  input for all meteorological applications - the
  number of some observation types exchanged per
  day is indicated)
• FM 12 SYNOP, FM 13 SHIP, FM 18 BUOY (53837
  synoptic pressure reports- from ECMWF Global Data
  Monitoring Report)
• FM 32 PILOT (188), FM 35 TEMP (1166) , FM 42
  AMDAR (17252), FM 63 BATHY, FM 64 TESAC, AIREP
  (4278)

4
The Table Driven Codes Forms (TDCF) BUFR/CREX

• Offer great advantages compared to the
  traditional alphanumeric codes (TAC)
• Self-description, flexibility, expandability,
  packing (BUFR), readability (CREX image of BUFR
  in alphanumeric code )
• For new parameters or new data types, no need to
  change software, just additional table entries
  (this is fundamental in light of the fast
  evolution of science and technology there are
  regular requests for representation of new data
  types, metadata, higher resolution (time and
  space) and higher accuracy)
• Expected increase in data quality and data
  quantity (e.g. coordinates transmitted with
  report, no more Volume A problems)
• However, they define the standard format of the
  physical data layout, they are not Application
  Program Interfaces

5
WMO decisions

• Approved as operational codes BUFR in 1988, CREX
  in 2000
• BUFR used for exchanging satellite, aircraft,
  radar, wind profiler, tropical cyclone data and
  archives
• CREX used for exchanging ozone, radiological,
  hydrological, tide gauge, soil temperature and
  tropical cyclone data
• The Commission for Basic Systems (CBS XII -
  December 2000) agreed to generate a plan for the
  migration from Traditional Alphanumeric Codes to
  Table Driven Codes
• CBS XII established an Expert Team on Migration
  to Table Driven Codes
• The Executive Council (EC- LIII, June 2001)
  endorsed these actions.

6
MIGRATION GOAL

• ULTIMATELY ALL OBSERVATIONS TO BE EXCHANGED IN
  BUFR (WHICH OFFERS MORE FEATURES THAN CREX, E.G.
  QUALITY FLAGS, ASSOCIATED VALUES, PACKING).
• FOR SOME COUNTRIES, HOWEVER, CREX CAN BE AN
  INTERIM SOLUTION.
• A BIG CHALLENGE!

7
CBS defined constraints for the MIGRATION to
Table Driven Codes BUFR/CREX

• CBS REQUESTED
• IT SHOULD BE A SMOOTH TRANSITION WITHOUT NEGATIVE
  IMPACTS ON THE WORLD WEATHER WATCH OPERATIONS
• NOT ALL AT ONCE, BUT PROGRESSIVE AND VOLUNTARILY,
  BY COUNTRY AND BY DATA TYPE
• CBS DEFINED THE BASIC PRINCIPLES OF A PLAN FOR
  THE MIGRATION
• THE PLAN SHOULD ALLOW FOR EVERY WMO MEMBER TO
  MIGRATE.
• ALL WMO MEMBERS STATES SHOULD HAVE THE FREEDOM TO
  SWITCH TO BUFR (OR CREX) WHEN THEY WANT AND
  WHEN THEY ARE READY TO DO SO
• THE MIGRATION WOULD BE A LONG-TERM PROCESS WITH
  CONSIDERABLE FLEXIBILITY.
• INCENTIVES FOR MEMBER STATES TO MIGRATE TO TABLE
  DRIVEN CODE FORMS SHOULD EXIST. THE BENEFITS OF
  THE MIGRATION HAVE TO BE CLEARLY EXPLAINED TO
  MEMBERS.
• MANUFACTURERS OF OBSERVING SYSTEMS AS WELL AS
  DEVELOPERS OF PROCESSING SOFTWARE (INCLUDING
  WORKSTATIONS) SHOULD BE MADE WELL AWARE OF THE
  PURPOSE OF THE MIGRATION AND OF ITS BENEFITS.

8
CONSIDER THE WORLD WEATHER WATCH DATA FLOW
9
WMO OBSERVATION DATA PRODUCERS(Global
Observation System)

• 185 NMCs PRODUCE TRADITIONAL OBSERVATIONS
• SYNOP, TEMP, PILOT
• A CERTAIN NUMBER OF SPECIALISED COLLECTING
  CENTRES PRODUCE
• SATELLITE DATA (THE MAJORITY ALREADY IN BUFR)
• AIRCRAFT DATA (AMDAR (SOME ALREADY IN BUFR),
  AIREP)
• SHIP DATA
• BUOY DATA
• XBT/CTD
• SUB-SURFACE PROFILING FLOATS DATA
• PRODUCERS WILL HAVE THE FREEDOM TO SWITCH TO BUFR
  WHEN THEY NEED (INTEREST OF SOME OF THEIR USERS
  FOR NEW PARAMETERS, DATA TYPES, METADATA).
  HOWEVER, ONE HAS TO INSURE THEIR OTHER USERS WILL
  STILL HAVE ACCESS TO THE DATA (ONE HAS TO ANALYSE
  DATA FLOW AND USER REQUIREMENTS)

10
The Global Observing System (GOS) and the
concept of double-dissemination for the migration
BUFR and CREX, or BUFR and TAC
11
Data conveyors 31 RTHs (GTS)
12
Impact of the migration for a Regional
Telecommunication Hub (RTH) which receives data
(still store and forward system)

• in Traditional Alphanumeric Codes (TAC) or CREX
• retransmits in TAC or CREX
• in BUFR
• retransmits in BUFR
• (and, if it has the capability, and it is
  necessary (some connected NMCs may have no binary
  capability, and there are no double dissemination
  from producer), it converts to CREX, then
  retransmits in CREX (double-transmission))
• Message Switching Directories will have to be
  updated (Same MSS for BUFR or CREX bulletins as
  for TAC. Hence, new bulletin header system to be
  finalized by WMO telecommunications experts)

13
Real time DATA USERS

• The Global Data Processing System
• 16 Centres running operationally global models
• 65 Centres running operationally regional or
  mesoscale models
• 185 (all NMCs) Centres performing meteorological
  applications (40 are not automated!)
• Who has a BUFR decoder? A universal one?
• Who has a CREX decoder? A universal one?
• First priority of the migration
• universal BUFR and CREX decoder for all!

14
GDPS Operational numerical models
15
The migration requires preparatory corrective
actions at GDPS Centres to avoid missing data

• Some Centres will have to add in their processing
  chain full universal BUFR and CREX decoders, to
  avoid missing observations.
• Other Centres will have to implement a universal
  BUFR and CREX decoders, to be able to receive
  observations in BUFR or CREX.
• Manufacturers of "turn key" work-stations
  inputting GTS data would need to be approached so
  that they include in their software universal
  BUFR and CREX decoders, either in the existing
  systems, or for the new systems currently in
  development or to be developed.
• Centres, which are currently operating manually
  (about 40 over 185), will have to seriously
  consider automation with software including a
  universal BUFR and CREX decoder. Before
  automation is implemented, or before they have a
  binary connection, they will be able to receive
  and understand CREX messages with a relatively
  simple training, and also encode and transmit
  observation reports in CREX.
• For some Centres the Internet could allow access
  to data in binary formats that are not available
  over their GTS link.

16
ACTIONS UNDERTAKEN

• WMO EXPERTS ON CODES HAVE DEFINED TEMPLATES FOR
  ALL TRADITIONAL OBSERVATIONS IN BUFR
• MANUFACTURERS OF AUTOMATIC WEATHER STATIONS AND
  OTHER OBSERVING PLATFORMS ALREADY INVITED TO TAKE
  INTO ACCOUNT THE COMING SHIFT TO TABLE DRIVEN
  CODES FOR THEIR SOFTWARE DEVELOPMENT.
• AS FROM NOVEMBER 2002, IN A VOLUNTARY AND
  EXPERIMENTAL MANNER, SOME DATA PRODUCERS, MAY
  TRANSMIT IN REAL TIME OBSERVATIONS IN BUFR OR
  CREX (AND ALSO IN TRADITIONAL ALPHANUMERICAL
  CODES, I.E. DOUBLE DISSEMINATION).
• THE WMO SECRETARIAT DISTRIBUTES GUIDANCE
  INFORMATION DESCRIBING IN A SIMPLE MANNER THE
  TABLE DRIVEN CODES AND EXPLAINING THEIR
  ADVANTAGES
• A NEW GUIDE ON BUFR/CREX WITH 3 LAYERS IS ON THE
  WMO WEB (http//www.wmo.ch/web/www/WDM/Guides/BUFR
  -CREX-guide.html)
• LAYER 1 FOR GENERAL PHILOSOPHY
• LAYER 2 FOR APPLICATION INTERFACING USERS, DATA
  AND TELECOMMUNICATION MANAGERS
• LAYER 3 FOR ENCODER/DECODER PROGRAMMERS
• NEW ANNEX TO MANUAL ON CODES DEFINING REPORTING
  PRACTICES ASSOCIATED TO BUFR AND CREX WILL BE
  WRITTEN IN 2002

17
Encoding/decoding

• Essential to have descriptors for all parameters
  that can be encoded in a traditional code form
  including regional and national practices
• And Templates layout of the format, list of
  descriptors to be transmitted (Common sequences)

18
ACTIONS EXPECTED FROM WMO MEMBER STATES

• DEFINE MIGRATION CONTACT POINTS (88 over 185 so
  far)
• IDENTIFY IMPACTS OF MIGRATION ON NATIONAL
  OPERATION
• NATIONAL TRAINING PROGRAMME ON TDCF
• SOFTWARE ADJUSTMENTS
• introduce BUFR/CREX encoder for the concerned
  data type at OBSERVATION, CONCENTRATION,
  DISSEMINATION
• process BUFR/CREX bulletins in MSS
• introduce BUFR/CREX process with decoder in
  PROCESSING CHAIN
• RECEPTION, DECODING, VISUALISATION, ARCHIVING
• NECESSARY FINANCIAL RESOURCES
• SEVERAL PRODUCERS WILL BE ABLE TO START TO
  TRANSMIT TRADITIONAL OBSERVATION DATA IN BUFR AT
  END OF 2002 OR IN 2003 (REPORTS FROM AUTOMATIC
  WEATHER STATIONS, RADIOSONDE STATIONS, IN SOME
  CASES CONVERSION AT CONCENTRATION SITE)

19
ACTIONS BY ASSOCIATED PROGRAMMES

• THERE ARE PRODUCERS AND USERS
• SATELLITE PRODUCERS PRODUCE ALREADY IN BUFR
• OTHER PRODUCING CENTRES WILL START TO SWITCH TO
  BUFR
• MARINE (JCOMM), NEED FOR BUFR ENCODERS, DOUBLE
  DISSEMINATION FOR USERS
• Service ARGOS will start dissemination in BUFR
  for buoys, XBT/CTD and sub-surface profiling
  floats in 2003
• ICAO ACARS TRANSMITTED IN BUFR, SIG WEATHER DATA
  TRANSMITTED IN BUFR, AMDAR SOON IN BUFR BUT LONG
  WAY FOR METAR/SPECI/TAF!
• USERS WILL HAVE TO BE PROVIDED WITH BUFR/CREX
  DECODERS

20
TENTATIVE MIGRATION SCHEDULE
21
MIGRATION WHAT ARE THE KEYS TO SUCCESS?

• TRAINING ON BUFR/CREX - WHAT IS IT?
• LEVEL 1 FOR GENERAL PHILOSOPHY
• LEVEL 2 FOR APPLICATION INTERFACING USERS, DATA
  AND TELECOMMUNICATION MANAGERS
• WMO SEMINARS
• TRAIN THE TRAINERS (WMO RMTCs, METEOROLOGICAL
  SCHOOLS, COLLEGES, UNIVERSITIES)
• SEMINARS IN COUNTRIES
• TRAINING WITHIN COUNTRIES NATIONAL TRAINING
  PROGRAMMES
• SOFTWARE PROJECT
• SOFTWARE HOUSE A CENTRE DISTRIBUTING FREE
  BUFR/CREX ENCODER/DECODER SOFTWARE AND PROVIDING
  ASSISTANCE FOR IMPLEMENTATION, DOCUMENTATION,
  ANSWERING QUERIES

22
SOFTWARE PROJECT REQUIREMENTS

• Build software to decode/encode BUFR, CREX (and
  GRIB 2). At a minimum, software should compile
  and run on the most common dialects of UNIX
  (including LINUX) and WINDOWS. It should be
  compatible with 32-bit and 64-bit architectures.
• The software delivered should be callable from
  applications, written in most common programming
  languages, such as FORTRAN and C.
• Printing and display routines should be available
  to view the whole contents of decoded data, and
  should be completed by user-friendly interface(s)
  as appropriate (JAVA or XML ON PC).
• The user should have access to Code tables, both
  for human reading, (and expressing requirements
  for additions if needed) and editing (new
  international additions or for local table
  entries).
• Documentation on the program clear and
  comprehensive, with all interfaces with the
  external application well defined.
• The software will be distributed to WMO Members
  and WMO associated Programs. It will be
  maintained and upgraded when required. Queries
  will be answered.
• Software (or relevant data type subset) should
  also be made available to manufacturers of
  systems providing or coding meteorological data.
• After the programming development is completed,
  the task is estimated for one person full time.

23
SOFTWARE HOUSE PROJECT

• IT IS A NEW PARADIGM FOR THE WMO BUT IT IS
  REQUIRED
• A CENTRALIZED UNIT DEFINING A STANDARD
  APPLICATION PROGRAM INTERFACE, AND DEVELOPING
  AND SUPPORTING APPLICATION PROGRAM INTERFACE
  SOFTWARE
• IT IS A VALUABLE AND NECESSARY STEP TO ENSURE
  THAT MODERN STANDARD DATA REPRESENTATION FORMS,
  DEVELOPED AND COORDINATED BY THE WMO, BE USED BY
  THE WIDEST POSSIBLE USER COMMUNITY
• THIS WILL BE OF PARTICULAR USE TO THOSE USERS
  WITH VERY LIMITED COMPUTER PROGRAMMING RESOURCES.

24
SOFTWARE HOUSE PROJECT

• A SOFTWARE HOUSE PROJECT IN EUROPE (ECMWF?,
  EUMETNET STATES?) WILL FAVOUR AND HELP MIGRATION
  TO TABLE DRIVEN CODES.
• REMARK THAT
• THE FIRST BENEFICIARY WILL BE THE EUROPEAN STATES
  THEMSELVES AND ALL THEIR METEOROLOGICAL
  APPLICATIONS (IN PARTICULAR) THEIR OPERATIONAL
  FORECAST, WITH THE PROSPECT OF RECEIVING MORE
  DATA AND BETTER QUALITY DATA.
• THANK YOU FOR YOUR ATTENTION!