The PEACE Database: Maintaining your SDDAS database access, data export and data visualisation tools R.J. Wilson and A.N. Fazakerley Mullard Space Science Laboratory Spring 2006 Version 1.1

1
The PEACE Database Maintaining your SDDAS
database access, data export and data
visualisation toolsR.J. Wilson and A.N.
Fazakerley Mullard Space Science
LaboratorySpring 2006Version 1.1
2
Content

• Introduction
• Overview of the activities
• Recommended approach
• Graphic how to guide
• Explanation of each activity

NoteThis document tells you how to update your
own local SDDAS installation, on your own
machine.If you use msslAB to run SDDAS then all
this is done for you.
3
Introduction (1)
The PEACE Database uses IDFS format files, and a
suite of software known as SDDAS is freely
available which enables you to work with the
database, export data and visualise the
data The Database changes with time as new data
is added and as calibrations change Processed
data in the Database (SPINPAD, moments,
spacecraft potential estimates etc.) may be
reprocessed from time to time Tools discussed
here allow you to check whether your local copy
of the database is up to date, and to make
updates when necessary Other tools allow you to
update the SDDAS software We also provide
suggestions on maintenance of your local copy of
the database
4
Introduction (2)
The Master Server for all PEACE data is at MSSL
(UK) Its IP number is currently 128.40.70.117 A
mirror copy of the PEACE database is at SwRI
(USA) Its IP number is currently 129.162.154.69
It is intended to be regularly updated to track
changes at MSSL
5
Overview of the activities

• Sync With Server
• Update your local copy of the Hierarchy from
  root server i.e. Tells your computer what
  products are available
• MegaMetaPromote
• Using your local copy of the Hierarchy,
  update your local copy of the data
  catalogue, from which you may promote data.
• Download Beta Apps Get the latest versions
  of SDDAS applications to plot/export data.
• Clearing out your promoted data area Stop
  your local Hard Disk filling up, and stops you
  using older data.

6
Recommended approach to maintenance

• Proposed Schedule and proposed order or
  execution
• Clearing out your promoted data areaAt least
  monthly for Cluster, weekly for Double
  Star (Weekly)
• Download Beta AppsAt least monthly (Month
  ly)
• Sync With ServerMonthly, or if you hear a new
  virtual has been added. (Monthly)
• MegaMetaPromote (always after Sync with
  Server)Weekly at least.If youre working with
  very recent data consider daily. (Daily)
• For reference, the main MSSL group computer is
  updated at the frequency in blue.

7
Caring for SDDASin pictures

• Select Admin Tools
• Sync with Server
• MegaMetaPromote
• Download Beta Apps

8
Select Admin Tools

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Config Files Editor
• Select Pull Down MenuAction gt Sync With
  Server
• From the list on the right, click the IP of the
  server you wish to sync with and the number
  appears on the left.Cluster/Double Star
  128.40.71.151Mars Express
  129.162.154.22Then select Ok.
• If files are already in sync it will tell you,
  otherwise you get
• You can select just parts the you want but
  easier to just select Select All and then OK
• You are returned to the earlier screen. Now you
  MUST use the pull down menuFile gt SaveDONT
  FORGET TO SAVE! (and make sure you have
  permissions to save the files)

9
Select Admin Tools

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Config Files Editor
• Select Pull Down MenuAction gt Sync With
  Server
• From the list on the right, click the IP of the
  server you wish to sync with and the number
  appears on the left.Cluster/Double Star
  128.40.71.151Mars Express
  129.162.154.22Then select Ok.
• If files are already in sync it will tell you,
  otherwise you get
• You can select just parts the you want but
  easier to just select Select All and then OK
• You are returned to the earlier screen. Now you
  MUST use the pull down menuFile gt SaveDONT
  FORGET TO SAVE! (and make sure you have
  permissions to save the files)

10
Sync with Server

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Config Files Editor
• Select Pull Down MenuAction gt Sync With
  Server
• From the list on the right, click the IP of the
  server you wish to sync with and the number
  appears on the left.Cluster/Double Star
  128.40.71.151Mars Express
  129.162.154.22Then select Ok.
• If files are already in sync it will tell you,
  otherwise you get
• You can select just parts the you want but
  easier to just select Select All and then OK
• You are returned to the earlier screen. Now you
  MUST use the pull down menuFile gt SaveDONT
  FORGET TO SAVE! (and make sure you have
  permissions to save the files)

11
Sync with Server

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Config Files Editor
• Select Pull Down MenuAction gt Sync With
  Server
• From the list on the right, click the IP of the
  server you wish to sync with and the number
  appears on the left.Cluster/Double Star
  128.40.71.151Mars Express
  129.162.154.22Then select Ok.
• If files are already in sync it will tell you,
  otherwise you get
• You can select just parts the you want but
  easier to just select Select All and then OK
• You are returned to the earlier screen. Now you
  MUST use the pull down menuFile gt SaveDONT
  FORGET TO SAVE! (and make sure you have
  permissions to save the files)

12
Sync with Server

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Config Files Editor
• Select Pull Down MenuAction gt Sync With
  Server
• From the list on the right, click the IP of the
  server you wish to sync with and the number
  appears on the left.Cluster/Double Star
  128.40.70.117
• Then select Ok.
• If files are already in sync it will tell you,
  otherwise you get
• You can select just parts the you want but
  easier to just select Select All and then OK
• You are returned to the earlier screen. Now you
  MUST use the pull down menuFile gt SaveDONT
  FORGET TO SAVE! (and make sure you have
  permissions to save the files)

13
Sync with Server
Material that is not up to date on your computer

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Config Files Editor
• Select Pull Down MenuAction gt Sync With
  Server
• From the list on the right, click the IP of the
  server you wish to sync with and the number
  appears on the left.Cluster/Double Star
  128.40.70.117
• Then select Ok.
• If files are already in sync it will tell you,
  otherwise you get a GUI like the one shown here
• You can highlight just parts the you want to
  update, and press OK or just select Select
  All and then OK
• You are returned to the earlier screen. Now you
  MUST use the pull down menuFile gt SaveDONT
  FORGET TO SAVE! (and make sure you have
  permissions to save the files)

Material that is no longer on the master
server,so can be deleted from your local system.
14
Sync with Server

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Config Files Editor
• Select Pull Down MenuAction gt Sync With
  Server
• From the list on the right, click the IP of the
  server you wish to sync with and the number
  appears on the left.Cluster/Double Star
  128.40.70.117
• Then select Ok.
• If files are already in sync it will tell you,
  otherwise you get a GUI like the one shown here
• You can highlight just parts the you want to
  update, and press OK or just select Select
  All and then OK
• You are returned to the earlier screen. Now you
  MUST use the pull down menuFile gt SaveDONT
  FORGET TO SAVE! (and make sure you have
  permissions to save the files)

When saving, it is normal to see a pop-up box
asking you to confirm that you want to overwrite
the current .cfg files
15
MegaMetaPromote
Always do this after, not before, Sync with
Server!

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Promote Meta Data
• Select Source
• You get a list of Source Definitions to click
  through. Easiest to just click the top level
  once, i.e. click CLUSTERII
• At his point just click Ok as you want all the
  Cluster spacecraft.
• This returns you to earlier window, with
  CLUSTERII in the Source at the top.Now just
  click Go! and wait for it to update.
• NB Do not Promote VIDFs as there are 100,000s of
  them.

16
MegaMetaPromote
Always do this after, not before, Sync with
Server!

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Promote Meta Data
• Select Source
• You get a list of Source Definitions to click
  through. Easiest to just click the top level
  once, i.e. click CLUSTERII
• At his point just click Ok as you want all the
  Cluster spacecraft.
• This returns you to earlier window, with
  CLUSTERII in the Source at the top.Now just
  click Go! and wait for it to update.
• NB Do not Promote VIDFs as there are 100,000s of
  them.

17
MegaMetaPromote
Always do this after, not before, Sync with
Server!

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Promote Meta Data
• Select Source
• You get a list of Source Definitions to click
  through. Easiest to just click the top level
  once, i.e. click CLUSTERII (and/or DOUBLESTAR)
• At this point just click Ok as you want all the
  Cluster spacecraft.
• This returns you to earlier window, with
  CLUSTERII in the Source at the top.Now just
  click Go! and wait for it to update.
• NB Do not Promote VIDFs as there are 100,000s of
  them.

18
MegaMetaPromote
Always do this after, not before, Sync with
Server!

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Promote Meta Data
• Select Source
• You get a list of Source Definitions to click
  through. Easiest to just click the top level
  once, i.e. click CLUSTERII (and/or DOUBLESTAR)
• At this point just click Ok as you want all the
  Cluster II spacecraft.
• This returns you to earlier window, with
  CLUSTERII in the Source at the top.Now just
  click Go! and wait for it to update.
• NB Do not Promote VIDFs as there are 100,000s of
  them.

19
MegaMetaPromote
Always do this after, not before, Sync with
Server!

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Promote Meta Data
• Select Source
• You get a list of Source Definitions to click
  through. Easiest to just click the top level
  once, i.e. click CLUSTERII (and/or DOUBLESTAR)
• At this point just click Ok as you want all the
  Cluster II spacecraft.
• This returns you to earlier window, with
  CLUSTERII in the Source at the top.Now just
  click Go! and wait for it to update.
• NB Do not Promote VIDFs as there are 100,000s of
  them.

20
Download Beta Apps

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Download BETA Apps
• Select Go
• Alternatively click the Updated Tab to see a list
  of files that have newer version at SWRI, then
  download them.

21
Download Beta Apps

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Download BETA Apps
• Either select Go, or
• Alternatively click the Updated Tab to see a list
  of files that have newer version at SWRI, then
  download them.

22
Download Beta Apps

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Download BETA Apps
• Either select Go, or
• Alternatively click the Updated Tab to see a list
  of files that have newer version at SWRI, then
  download the ones you want.

23
Download Beta Apps

• At console prompt, typesdcontrol
• Select Administration Tools
• Select Download BETA Apps
• Either select Go, or
• Alternatively click the Updated Tab to see a list
  of files that have newer version at SWRI, then
  download the ones you want.
• To finish now select Quit.

24
Clearing out your promoted data area

• When you promote data (or VIDFs, PIDFs) they go
  in to the hierarchy subdirectories such as those
  beneath /home/sddas/data/CLUSTERII (for
  Project CLUSTERII)
• /home/sddas/data/DOUBLESTAR (for Project
  DOUBLESTAR)
• These files stay there and take up room on your
  hard disk.
• i.e. a single sensor Cluster PEACE data file
  (D) from PAD may be 10 Mb in size If want both
  HEEA LEEA for a short period (hours), over all
  4 spacecraft thats 80 Mb.
• A single day Double Star PEACE 3DF data file (D)
  can easily be 500 Mb,If you want both TC-1 and
  TC-2 then thats 1000Mb, or 1 Gb.
• You local Hard Disk will quickly become full if
  you do not delete these files after use.This can
  then stop your entire computer working properly
  due to lack of disk space.
• Delete files you no longer need!!!
• Files to delete are

25
Clearing out your promoted data area

• PIDFS Only small files (80k), relatively few
  of them.
• VIDFS Only small files (80k) but many.
• Header files Small-ish files (up to a few Mb)
  but many.
• Data files LARGE files (10s to100s of Mb) and
  many of them.
• Why delete the small (or large) files at all?
  Maybe theyve been updated on the master server
  with new calibrations or re-processed. Deleting
  them ensures you promoted the latest version next
  time you use them.
• A brute force method of deleting all promoted
  files, for project CLUSTERII, assuming your local
  data directory is in /home/sddas/data/ rm
  /home/sddas/data/CLUSTERII////.pidf.v2 -
  delete PIDFsrm /home/sddas/data/CLUSTERII////
  V.v3 - delete version 3 VIDFsrm
  /home/sddas/data/CLUSTERII////IV -
  delete version 2 VIDFs, both file typesrm
  /home/sddas/data/CLUSTERII////HD -
  delete Header and Data filesSimilar for projects
  DOUBLESTAR or MARS.
• Do not delete anything from the Database
  directories, i.e. leave /home/sddas/data/CLUSTERII
  ///Database alone.The use of wildcards
  illustrated above is safe as no files in the
  Database directory match the ending letters
  above.
• (but if you do after all delete from the
  Database directories you can recover using
  MegaMetaPromote)

26
Explanation of the Activities
27
Anatomy of IDFS 1

• In IDFS you have a hierarchyfrom mission down to
  instrument
• Project gt Mission gt Experiment gt Instrument
• i.e.
• CLUSTERII gt CLUSTER-1 gt PEACE gt 3DF
  
• At the Instrument level you have virtual
  instruments, which are subsetsof the
  instruments data.
• i.e. CP3DFH and CP3DFL
• virtuals have a set naming convention
  (Project, experiment, description)
• CP3DFH Cluster PEACE 3DF HEEA
• DP3DF DoubleStar PEACE 3DF (single sensor)
• CFUNIT Cluster FGM UNIT vector
• DSPRIME DoubleStar Staff PRIME parameters
• Name must be unique and no more than 8
  characters.
• DHIAPRIM DoubleStar HIA PRIME parameters
• DHIDPRIM DoubleStar HID PRIME parameters
• (Exception to convention SPINAD with virtuals
  such as HSPAD LSPAD)

28
Anatomy of IDFS 2

• There are 4 types of file, VIDF, PIDF, Header and
  Data
• PIDF Plot Interface Description File.
• Just one per virtual, contains labels and unit
  choices.
• Filenames of virtual.pidf.v2 - text file
  that you can read
• i.e. CPPADH.pidf.v2
• VIDF Virtual Instrument Description File.
• Contains calibrations and conversions for units.
• Can be many VIDFs per virtual, each file contains
  a start and stop time inside it.
• Filename contains start time (to nearest minute).
• VIDFS must not overlap in period.
• virtualstart-timeV - version2 VIDF text
  file (for human use)
• virtualstart-timeI - version2 VIDF binary
  file (for SDDAS use)
• virtualstart-timeV.v3 version3 VIDF text
  file (for human SDDAS use)
• Start-time in format (year)(doy)hhmm, ie.
  20050381400 for 2005-Feb-07 1400
• i.e. CPPADH20050381400V.v3

29
Sync With Server

• Updates your machine with the root server that
  contains the latest version of the Hierarchy.
  i.e. including any new virtuals recently added.
• Modifies
• /usr/local/sddas/config/DB.cfgContains list of
  what servers you can promote from for each
  Project. (IP addresses listed rather than server
  names.)
• /usr/local/sddas/config/PROJECTS.cfgContains
  list of full hierarchy for each Project.
• Doing a Sync with Server updates these files for
  you.It shows differences only between your
  current copy and the master server.Makes sure
  you have permissions to write/save these files,
  if not use the chmod command or run as superuser.
• You need to know the IP address of the server you
  wish to Sync to
• CLUSTERII/DOUBLESTAR IP at MSSL
• 128.40.70.117

30
MegaMetaPromote

• In your local SDDAS Data directory (e.g. here we
  assume it is/home/sddas/data/ ?) you have your
  Hierarchy. Under each experiment you will have a
  Database directory, i.e. /home/sddas/data/CLUSTERI
  I/CLUSTER-1/PEACE/Database
• Within the Database directory, each virtual has 4
  files in two pairs
• virtual.HD.DBF virtual.HD.NDX - Header Data
  list
• virtual.I.DBF virtual.I.NDX - VIDF list
• These contain what your computer thinks is
  available for promotion,as opposed to what is
  really available for promotion from the master
  server.
• Do a MegaMetaPromote to update this list from the
  master server.Makes sure you have permissions
  to write/save these files in ALL the Database
  directories, if not use the chmod command or run
  as superuser.
• This is essential as you can only promote the
  data that your computer thinks is available...

31
Download Beta Apps

• SDDAS is made up from a number of separate
  programs ( 80!)
• To ensure you have the latest version of each you
  Download Beta Apps
• Download Beta Apps updates older versions of
  programs with newer ones from Southwest Research
  Institute (SwRI).
• Older versions are copied to /usr/local/sddas/back
  ups rather than being over-written. Should the
  new version be worse (i.e. new bugs) than the
  last you can then restore the old one from the
  backups directory.Makes sure you have
  permissions to write/save these files in the
  /usr/local/sddas/backups directory, if not use
  the chmod command or run as superuser.
• Do not use Download Applications this is
  rarely updated by SWRI,Always use the Beta Apps.