Status of HPSS

1
Status of HPSS
New Features, Requirements, and Installations
Otis Graf IBM Global Services - Federal Houston,
Texas October 1999
2
Topics

• HPSS New Features since Release 3.2
• Next incremental updates
• Feature to be added in 4.2
• Status of HPSS Installations
• Focus on Support Services
• Next Requirements Development Cycle

3
Summary of HPSS Releases and Patches
4
Features Added Since Rel. 3.2

• Includes through 4.1.1 (July 1999)
• File families
• Segregate files onto different groups of tapes.
• Storage Server uses family id passed in segment
  create calls for selecting the tape for the
  segment create

5
Features Added Since Rel. 3.2 (Cont.)

• Filsets A hierarchical collection of files and
  directories managed as a single unit.

fs_goodwinj
fs_teaff
\
\
work
play
games
play
6
Features Added Since Rel. 3.2 (Cont.)

• HPSS Filesets
• Managed by name server
• Can be either HPSS-only or HPSS/DFS
• DFS Filesets
• DFS-only
• HPSS/DFS archived
• HPSS DFS name spaces not kept in synch
• Only way to get data is through DFS
• Performance similar to DFS-only if data is
  resident
• HPSS/DFS mirrored
• HPSS DFS name spaces kept in synch
• Data can be seen and altered from either DFS or
  HPSS
• HPSS high speed data movement can be used

7
Features Added Since Rel. 3.2 (Cont.)

• PFTP Enhancements
• Allows the use of multiple network and/or
  multiple systems for child processes for
  multi-network stripe transfers
• MPI-IO Interface
• Implements a subset of the MPI-2 standard
• Coordinates access to HPSS files from multiple
  processes

8
Features Added Since Rel. 3.2 (Cont.)

• Scaleable Accounting
• Accounting summary records support the following
  collection options during the accounting period
• By account index COS,
• Number of accesses
• Number of files transferred
• bytes used
• By account index, COS, storage class
• number of files transferred
• number of accesses

9
Features Added Since Rel. 3.2 (Cont.)

• Performance Enhancements
• Support disk partitions gt 2 GB
• Bigger virtual volumes (up to 16,384 segments per
  VV)
• Faster disk file creates and purges
• Support for Shelf Tapes
• Provide utility to identify and move volumes to
  shelf
• Mark cartridge metadata with shelf descriptor
• Generate operator mount requests via SSM pop-up
• Client API Enhancements
• Manually migrate/purge on a file basis
• Lock/unlock to prevent a file from being purged

10
Features Added Since Rel. 3.2 (Cont.)

• Non-DCE client API (Solaris, IRIX AIX)
• Non-DCE Mover
• Mover ported to other platforms
• Solaris (DCE)
• SGI IRIX (non-DCE)
• Additional Tape Drive Support
• Ampex DST 314
• IBM 3590E
• Additional Library Support
• ADIC Automated Media Library (AML)

11
Number of HPSS Systems

• Notes
• System is a name space instance intended for
  production
• 1999 data is year to date.

12
Listing of HPSS Sitesas of October 1999
13
Listing of HPSS Sites (Cont.)as of October, 1999
14
Listing of HPSS Sites (Cont.)as of October, 1999
15
Current Activities

• Next scheduled patch (4.1.1.2 in Nov 99)
• Sun Phase 2
• STK PVR and IBM 3494 PVR
• PFTP Client and Daemon
• MPI-IO
• SFS Backup utilities
• Some manageability improvements
• Follow-on patch (first quarter 00)
• IBM 3590 tape drives on Solaris
• StorageTek RAIT - testing and prototyping
  activity, no HPSS software change

16
Current Activities (Cont.)

• Development of Release 4.2
• Planned date is 4th Quarter 00
• Enhancement of On-Line Resources
• Tools repository
• On-line help (FAQ problem/solution DB)
• On-line documentation (HTML PDF)
• New Training Classes
• Advanced System Admin Class
• HPSS API programming class

17
Current Activities (Cont.)

• Enhancement of Customer Support
• Better coordination with IBM Transarc
• Deployment/support procedures

18
Features in Rel. 4.2

• Multiple Storage Subsystems
• Comprised of Name Server, Bitfile Server,
  Migration Purge Server, 1-n Storage Servers
• May be used to enhance concurrency
• May be used to partition servers to handle parts
  of the name space Data assigned to Storage
  Subsystem based on name
• Fileset creation established association with
  Name Server
• Junction attached fileset to name space

Note HPSS Contract terms are pending for
multiple name spaces.
19
Features in Rel. 4.2 (Cont.)

• Federated Name Space
• Defined as cooperative, distributed HPSS systems
  linked at the internal server level
• Supports use of all HPSS interfaces across
  multiple HPSS sites
• Maintains autonomy of individual HPSS
  installations
• Name spaces linked by junctions which point to
  filesets

20
Features in Rel. 4.2 (Cont.)

• Command Line Utilities
• Provide command line interface to SSM to enable
  management of HPSS from automated scripts
• Command-line program (hpssadm) which can be run
  interactively or in batch mode
• Allows user to manage servers, devices, storage
  classes, PVL jobs, and, to a limited extent,
  volumes

21
Features in Rel. 4.2 (Cont.)

• Non-DCE Client API Security
• Note The majority of the non-DCE Client API has
  been developed for released in Release 4.1.1
• Support client authorization / authentication by
  
• None
• DCE (for platforms with DCE)
• Kerberos
• Provide Non-DCE Client Gateway (NDCG) with the
  ability to authenticate clients identities on a
  per connection basis

22
Features in Rel. 4.2 (Cont.)

• Mass Configuration
• Provide mechanisms to create multiple server or
  drive / device records in a single request
• Provide mechanism to set persistent default
  values used in creating new configuration records
• A configurable default Log Policy will be
  provided, which may reduce the number of specific
  policies which need to be created

23
Features in Rel. 4.2 (Cont.)

• Gatekeeper
• Provides optional client interface to allow sites
  to schedule and monitor use of HPSS file
  resources
• Supports file create, open, stage, and close
  requests
• May be associated with one or many Storage
  Subsystems
• Sites will be able to implement site specific
  user level scheduling of storage requests. This
  may include
• Limit the number of open files per user/host
• Prevent create requests for a user/host
• Collect statistics

24
Features in Rel. 4.2 (Cont.)

• Account Validation
• Maintain integrity of user account information.
  Users will only be able to use valid accounts
• Support cross-linked namespaces (remote sites).
  Individual sites select their own style of
  accounting
• While still supported, no longer be necessary to
  have user default account information in DCE
  registry
• Sites that do not need accounting or validation
  of user accounts may continue to run in the
  current fashion

25
Features in Rel. 4.2 (Cont.)

• Sun Solaris Phase 3
• Port remaining servers
• Target OS is Solaris 8
• HPSS back-end the IBM General Parallel File
  System (GPFS)

26
Post Release 4.2 Discussions

• No definitive release plans yet
• Features that were delayed from Rel. 4.2
• Tape import/export
• Multiple distributed movers - allow multiple,
  possibly distributed movers to access a single
  device
• Begin new requirements development cycle 2 Qtr
  00
• Features under discussion
• Improved manageability features
• Steps towards server consolidation and decreasing
  SFS dependencies
• Improved GPFS/HPSS performance through parallel
  data paths
• Backup/archive feature (file servers,
  workstations)
• Critical HEP requirements

27
Next Requirements Development Cycle

• Process
• Process begins at start of a new release cycle.
• Requirements contributed by customer and IBM.
• Requirements team consists of a representative
  from each customer site.
• Release contents are prioritized by requirements
  team.
• Developers responsible decomposing requirements
  into subsystem level.

28
Requirements Development Cycle (Cont.)

• Process (Cont.)
• Requirements are costed, and then reprioritized
  as an iterative process. Customer balloting
  influences release contents, but is not binding.
• Project metrics used to determine the number of
  requirements accepted in the next baseline.
• Requirements not making baseline are included in
  the requirements document, but annotated as
  future release.
• Formal inspection of requirements is performed.

29
Requirements Development Cycle (Cont.)

• Process (Cont.)
• Any changes to the approved baseline require
  Technical and Executive Committee approval.
• Subsystem Requirements documents are published
  once a baseline is approved.
• Opportunity for the HEP Community
• HEP community develop common set of requirements
• Provide unified voice on requirements committee

30
Additional Resources

• HPSS User Forum Presentations
• www5.clearlake.ibm.com ? Whats New
• Site Reports
• Details for current capabilities
• Details of Release 4.2
• Details of DFSS and HPSS
• Multi-Platform Capability of HPSS Components
• www5.clearlake.ibm.com ? Product Information
  ? Multi-Platform Capability of HPSS
• Core Servers and Movers
• User Interface Clients